notes from /dev/null - org mode

Announcing Casual Org

2026-02-19T15:00:00-08:00

Most Emacs Org users would concur that Org mode is a magnificent tool for capturing and communicating thought. That said, Org mode’s vast set of features can be daunting to master. A common guidance for new users is to take it slow: incrementally learn a subset of Org’s features as you need them.

A big reason for Org mode’s steep learning curve is that it adopts Emacs’ unfortunate culture of compelling users to memorize keybindings. Learning a distinct keybinding for each Org command (remember I said vast feature set?) is onerous, so a different tack is made: reuse the same keybinding but have it possess different behavior based on context. This context is usually tied to the type of structure the point (aka cursor) is in. For example, if the point is in a source block, the binding C-c C-c would execute it, but if the point is on a checkbox item, then C-c C-c would toggle its checked state. Taking this approach lowers the effort to recall a keybinding at the cost of recalling what its contextual behavior would be. In practice, using such overloaded keybindings is…okay. But I’d argue that we could have a more usable interface, leading to the point of this post:

Announcing support for Org mode in the v2.14.0 update for Casual, now available on MELPA.

Readers of this blog will know that Casual is my project to re-imagine the primary user interface for Emacs using keyboard-driven menus. If this is new to you, I highly recommend reading this introduction to it.

Primary to the design of the Casual Org menus is to be context-sensitive, only showing a subset of Org mode commands that are relevant. This set of context-sensitive commands is opinionated, but seeks to provide utility to users both new and experienced with Org.

Shown below is a demo of Casual Org at work:

Your browser does not support HTML video. Link to video.

While the design of this UI has been months in the making, there is nothing like real-world use and feedback. Constructive input is appreciated, especially if you are relatively new to using Org.

A great deal of thanks goes out to the maintainers of and contributors to Org mode and to the community that uses it. If you are able, please support it. Also, if you find Casual useful, I’d appreciate a coffee as well.

—

Video music: Burna Boy - Last Last (JON REYES FINDAWAY BLEND) | Jon Reyes

Export Org to Markdown with the Clipboard

2025-12-17T11:30:00-08:00

In symmetry with my post, Import Markdown to Org with the Clipboard in Emacs, exporting Org to Markdown via the system clipboard might be desired instead. Turns out, Marcin Borkowski has already done this as detailed in his post Org-mode to Markdown via the clipboard with the function org-copy-region-as-markdown. In my Emacs initialization, I’ve taken the liberty of prefixing that function name with mb/ to disambiguate it with actual functions from Org.

If you prefer to write in Org but need to post in Markdown, definitely give Borkowski’s post a look.

Import Markdown to Org with the Clipboard in Emacs

2025-12-04T13:30:00-08:00

The preponderance of Markdown formatted text in both websites and apps is a blessing and a point of friction for Org users. The content is there, but if you’re an Org user, you really prefer the text to be formatted for Org. If you have Pandoc installed, converting from Markdown to Org is trivial but laborious as it involves making the right command invocation with temporary files.

Thankfully, Emacs can automate this conversion as described in the workflow below:

From the source app or website, copy the Markdown into the system clipboard. This step pushes this text into the Emacs kill-ring.
In Emacs, invoke a command which converts the above Markdown text into Org, then pastes (yanks) converted text into a desired Org file.

Note that this workflow presumes that the Emacs kill-ring is integrated with the system clipboard and that pandoc is installed.

The following Elisp command cc/yank-markdown-as-org implements the above workflow:

(defun cc/yank-markdown-as-org ()
  "Yank Markdown text as Org.

This command will convert Markdown text in the top of the `kill-ring'
and convert it to Org using the pandoc utility."
  (interactive)
  (save-excursion
    (with-temp-buffer
      (yank)
      (shell-command-on-region
       (point-min) (point-max)
       "pandoc -f markdown -t org --wrap=preserve" t t)
      (kill-region (point-min) (point-max)))
    (yank)))

This command can be invoked in numerous ways including by key binding, context menu, Transient menu, and directly via M-x. My preference is using the context menu as illustrated in the screenshot below to optimize for mouse copy/paste interactions between different apps.

Users interested in seeing how I’ve configured my context menu can read the source here. Users unfamiliar with Emacs context menus can learn more about it in my post Customizing the Emacs Context Menu.

Closing Thoughts

Implementing cc/yank-markdown-as-org turned out to be a lot more simpler than expected, as the command shell-command-on-region does the heavy lifting in abstracting away the pandoc command invocation and its handling of temporary files. This implementation can serve as a example for other conversions using the kill-ring.

A word of thanks to mekeor on IRC for reminding me that (point-min) and (point-max) exist.

Storing a Link from your Web Browser to BibTeX using Org protocol

2025-09-25T16:15:00-07:00

If you’re familiar with this blog, you’ll know that I’m a fan of Org protocol, a means of sending data such as web links and content to an Org file from outside of Emacs. Following up on my last post on Casual BibTeX, this post will show how to re-purpose Org protocol to send a link from your preferred web browser to a BibTeX file.

This is achieved by adding the following template (named “bib”) to the variable org-capture-templates as shown below:

(add-to-list org-capture-templates
             '("bib"
               "BibTex Online Entry"
               plain
               (file "~/org/bib/references.bib")
               (function (lambda ()
                           (string-join
                            (list "@Online {,"
                                  "author = {%^{Author(s)}},"
                                  "organization = {%^{Organization}},"
                                  "title = {%:description},"
                                  "url = {%:link},"
                                  "date = {%<%Y-%m-%d>%?},"
                                  "notes = {"
                                  "%i"
                                  "}"
                                  "}")
                            "\n")))
               :prepend t
               :empty-lines-after 1))

The web link is treated as an BibTeX Online entry type whose fields are populated as follows:

url field holds the web page link.
title field holds the web page title.
date is populated with the current day (this should be edited to the actual date).
notes is populated with the body value according to the capture protocol.
author is populated with the value entered via mini-buffer prompt.
organization is populated with the value entered via mini-buffer prompt, typically the name of the website.

The template is configured to prepend (via the :prepend property) the captured Online entry to the file “~/org/bib/references.bib”. These values can be adjusted to taste.

When invoking this template via Org protocol, a temporary capture window holding the populated template is created. This entry will not have a citation key, so it is recommended to run M-x bibtex-clean-entry to generate it before committing the capture via C-c C-c. If you have Casual BibTeX installed, you can invoke the Transient menu casual-bibtex-tmenu with the binding M-m and invoke the clean command from there.

Closing Thoughts

Arguably, Org protocol was not designed to support files other than those using Org markup, so the above practice skirts its design intent by sending the capture to a BibTeX file. But as of Emacs 30.2 and Org 9.7.34, this works fine, so YOLO. Users intent on customizing the above “bib” template should be advised that many template properties are not supported, so be advised.

That all said, it’s a nice win to automate the task of writing a BibTeX Online entry.

Capturing Org Source Blocks

2025-07-08T17:00:00-07:00

A commonplace activity is to copy source code from a web page into an Org file. Ideally such code is kept structurally intact by wrapping it in an Org source block. This is typically done as follows:

Select source code text in web page.
Copy source code text.
Paste source code into Org file.
Annotate the pasted source code with an Org source block, specifying language. (#+BEGIN_SRC {language}, #+END_SRC)
- Either done manually or using the command org-insert-structure-template (C-c C-,)

While the above is not complicated, it can get onerous if done frequently. Thankfully Org provides you with the means to automate this. For example with Org protocol, the above steps can be reduced to:

Select source code text in web page (no need to copy!).
Run web bookmark to capture selected text to Emacs.
Select source code language via mini-buffer completion in Emacs and commit capture (C-c C-c).

Interested? Read on. Some Elisp ahead, but hopefully not too challenging.

Capturing code with Org protocol

A convenient way to capture content from a web page is through Org protocol which builds on top of Org capture. Some setup is required before using it, but the effort is rewarded in saved time for a future you.

Shown below is a capture template with key name “code”.

("code"
 "Source Code (Org Protocol)"
 entry
 (file "~/org/notes.org")
 (function (lambda ()
             (string-join
              (list "* Source: %:description"
                    ":PROPERTIES:"
                    ":CREATED: %U"
                    ":END:"
                    "%:link"
                    (concat "#+BEGIN_SRC "
                            (cc-org-capture--code-choices "elisp"))
                    "%i"
                    "#+END_SRC"
                    ""
                    "%?")
              "\n")))
 :empty-lines 1)

The Elisp function cc-org-capture--code-choices will generate capture expansion markup to choose a language with mini-buffer completion. It is configured here to use “elisp” as the default language. The set of languages to choose from via completion is defined in cc-org-capture--src-languages. Both their definitions are shown below:

(defun cc-org-capture--code-choices (lang)
  "Create selection of programming language choices with default LANG."
  (concat
   "%^{Language|"
   lang
   "|"
   (string-join cc-org-capture--src-languages "|")
   "}"))

(defvar cc-org-capture--src-languages
  (list
   "C" "F90" "R" "awk" "clojure" "cpp" "css" "ditaa" "dot" "elisp" "eshell"
   "forth" "gnuplot" "haskell" "java" "js" "julia" "kotlin" "latex" "lisp" "lua"
   "makefile" "matlab" "max" "ocaml" "octave" "org" "perl" "plantuml"
   "processing" "python" "ruby" "sass" "scheme" "sed" "shell" "sql" "sqlite"
   "swift" "swiftui" "tcl")
  "List of supported Org capture languages.")

To exercise the capture template named “code” from your JavaScript enabled web-browser, create a bookmark with the following JavaScript code as its location. When you find code you want to capture, select the code and run this bookmark.

javascript:location.href='org-protocol://capture?' +
      new URLSearchParams({
            template: 'code', url: window.location.href,
            title: document.title, body: window.getSelection()});
      void(0);

Aside: if you use Emacs for macOS, you’ll want Scrim to get Org protocol to work.

Capturing source code by copying

Often it is desirable to just copy the source code (putting it in the kill-ring) and paste (or rather capture) it direct into an Org (or Markdown) file of one’s choosing. The following capture template named “cc” can accomplish this.

("cc"
 "Code"
 plain
 (here)
 (function cc-org-capture--code-select-body)
 :empty-lines 1
 :immediate-finish 1)

In the above template, the target (here) is used to insert the captured source code at the current point. The template type is set to plain to not give the captured source code any special treatment with respect to Org structure. The :immediate-finish key is set to non-nil to not create a separate capture window.

The “cc” template is intended to be invoked with a multiple-key sequence starting with ’c’, so the following entry in org-capture-templates is expected.

("c" "Code")

The work of constructing the wrapped source code (in this case, presumed to be in the kill-ring via system copy) is done in cc-org-capture--code-select-body. Its source is shown below.

(defun cc-org-capture--code-select-body ()
  "Body capture code in `kill-ring' head with language prompt."
  (cc-org-capture--code-body-from-kill-ring
   (cc-org-capture--code-choices "elisp")))

(defun cc-org-capture--code-body-from-kill-ring (lang)
  "Generate capture body for LANG code at head of the `kill-ring'."
  (string-join
   (list
    (cc-org-capture--src-block-begin lang)
    (if kill-ring "%c" "")
    (cc-org-capture--src-block-end))
   "\n"))

(defun cc-org-capture--src-block-begin (&optional lang mode)
  "Annotate begin of source code block for given LANG and MODE."
  (let* ((lang (if lang lang "elisp"))
         (mode (if mode mode major-mode)))
    (cond
     ((eq (derived-mode-p mode) 'org-mode) (concat "#+BEGIN_SRC" " " lang))
     ((eq (derived-mode-p mode) 'markdown-mode) (concat "```" lang))
     (t (concat "#+BEGIN_SRC" " " lang)))))

(defun cc-org-capture--src-block-end (&optional mode)
  "Annotate end of source code block for given MODE."
  (let* ((mode (if mode mode major-mode)))
    (cond
     ((eq (derived-mode-p mode) 'org-mode) "#+END_SRC")
     ((eq (derived-mode-p mode) 'markdown-mode) "```")
     (t "#+END_SRC"))))

The implementation of cc-org-capture--code-select-body will check the current major mode (major-mode) and depending if it is Org or Markdown, wrap the copied source code in the kill-ring (expanded by the term “%c”) accordingly.

The following code shows how the “cc” template can be added to org-capture-templates.

(add-hook
 'org-mode-hook
 (lambda ()
   (add-to-list 'org-capture-templates
                '("cc"
                  "Code"
                  plain
                  (here)
                  (function cc-org-capture--code-select-body)
                  :empty-lines 1
                  :immediate-finish 1)
                t)))

If you have C-c c bound to org-capture then you should see the “cc” template in the ’c’ sub-menu.

Caveats

If the source code to be captured contains any text that matches a capture template expansion then it will be processed, likely resulting in unintended behavior. Look out for text that starts with ’%’.

Closing Thoughts

If you’re still here thanks for your attention! This post required a fair amount of Elisp understanding, but the reward for knowing it is the ability to build very bespoke Org capture workflows. Motivated readers are encouraged to take these examples and run with them. To this end I’ve deliberately left out a capture template example of “hard-coding” a language as an exercise to the reader.

If you end up making a template that you think others would find useful, please share it!

Links

If you're on macOS and have Org protocol setup, you might be interested in this post "Capturing an Org note via macOS Shortcuts".

Capturing an Org note via macOS Shortcuts

2025-07-02T14:30:00-07:00

With a new update of Scrim v1.1.0 on the App Store, it seems an opportune time to show a nice macOS Shortcut integration with it.

The use case is this: You're on your Mac, working on something else other than Emacs (it happens!) and you want to make a quick note in an Org file without having to context switch to Emacs.

You can accomplish this with a macOS Shortcut. Shortcuts is an Apple tool that lets you orchestrate different apps to achieve a custom workflow.

Shown below is the shortcut named “Make Note Entry” to accomplish the above use case. It will:

Prompt the user to enter text.
Run a Python script on that text and construct an Org protocol capture request with it using the template named "note".
Issue (open) the Org protocol request.

If the shortcut is configured so that “Pin in Menu Bar” is turned on, you will see some variant of its entry in the Shortcuts bar icon menu.

Running the shortcut “Make Note Entry” will prompt you with a dialog where you can enter your note:

Pressing the “Done” button will result in the entry being inserted at the top of the file “~/org/notes.org” as shown below. Note however that Emacs is not raised with this shortcut. The screenshot below is only shown to illustrate that the entry is there.

Installation

Want to try this out? Here’s what to do:

1. Understand Org protocol

If you don’t already then read up about it at org-protocol.el – Trigger actions in Emacs via a custom URL scheme.

2. Have Scrim installed and configured

Learn more about Scrim at http://yummymelon.com/scrim.

3. Add the capture template “note”

On the Emacs side, add the following template entry to org-capture-templates. This template named “note” defines an entry with a heading that has an inactive Org timestamp (“%U”) and the body containing the entered text (“%i”). The destination of this note entry is in the file “~/org/notes.org”

("note"
 "Note (Org Protocol)"
 entry
 (file "~/org/notes.org")
 (function (lambda ()
             (string-join
              '("* %U"
                "%i")
              "\n")))
 :prepend t
 :immediate-finish t
 :empty-lines-after 1)

Observe that this template uses the keys :immediate-finish to not make the entry interactive and :prepend to insert the entry at the top of the file.

4. Add the Shortcut “Make Note Entry”

Download the shortcut file Make Note Entry.shortcut and double-click on it to install. It should now be in your Shortcuts collection.

5. Run “Make Note Entry”

On the first run of this shortcut, you will be prompted twice to give it permissions: 1) to run the shortcut and 2) to open Scrim. Inspect that its contents correspond with its screenshot above and grant them.

Closing Thoughts

Readers are invited to take these ideas and customize them for their own purposes.

Announcing Scrim - An Org Protocol Proxy for Emacs on macOS

2025-04-28T05:00:00-07:00

Org mode celebrates its ability to export to a multitude of formats. However, lesser known is its ability to import content. One of the ways to import into Org is through a custom URL scheme. This scheme is referred to as Org Protocol (scheme name: org‑protocol://). The motivating idea for using a URL scheme was this: any app outside of Emacs could construct and make an org‑protocol:// URL request without needing any security permissions.

Regrettably, the original implementation of Org Protocol presumed that emacsclient would do the work of making the actual URL request to Emacs. This thinking did not consider the notion that app sandboxing would become widespread practice, preventing other programs from calling emacsclient. So it is with recent releases of macOS: its default policy is to only allow an app to access programs within its prescribed sandbox.

That said, I ❤️ Org Protocol, so much so that I provided support for it in a utility called Captee that lets you share links and text clippings from the macOS Share Menu. I wanted Org Protocol to work on macOS, but for years it was broken due to its dependence on emacsclient. The only workaround was to use the Mitsuharu Yamamoto fork of Emacs which implemented native support for org-protocol:// URL requests. For a number of years, the Yamamoto fork was my daily driver. Unfortunately though, it seems to have succumbed to bit-rot as it has not kept track with recent releases of GNU Emacs (in particular 30.1). I made the switch over to GNU Emacs 30.1, but in doing so I had to give up Org Protocol support.

I missed Org Protocol, so I decided to do something about it. I wrote an app.

Scrim is a macOS utility to act as a proxy to take an Org Protocol request from any app and relay that to Emacs. In effect, Scrim is a native macOS client for Emacs server.

Learn more about Scrim at its website.

Scrim is now available for purchase on the Mac App Store.

As Scrim and Captee compliment each other so well, you can get them both at a discounted price.

Making an App, Looking for Beta Testers

2025-04-14T14:50:00-07:00

Nearly two years back I wrote a utility called Captee that uses the native macOS Share Menu ( ) to share a link or text clipping in either Markdown or Org format. The idea was that any macOS app that supported the Share Menu could take advantage of this. Among the features that Captee supported was the ability to share content via Org Protocol, a custom URL scheme devised to import content into Emacs.

However, for Org Protocol to work, it required that there be a way to transport the org-protocol:// URL request to Emacs on macOS. Org Protocol’s original implementation presumed that emacsclient would do the lifting here, where the sharing app would be able to call emacsclient with the org-protocol// request as its argument. This was however a regrettable design decision made over a decade ago as recent versions of macOS now have a default sandbox policy that prevents an app from calling an executable like emacsclient outside of its sandbox.

The Mitsuharu Yamamoto fork of Emacs got around this restriction by supporting the Org Protocol custom URL scheme natively, where it would directly accept the URL request, bypassing emacsclient. Unfortunately for users of GNU FSF-published Emacs, Org Protocol was viable only if emacsclient was enabled, typically by relaxing macOS security permissions.

To address the above for GNU FSF-published Emacs, I set out to build a proxy utility that would accept the org-protocol:// URL request and transport it to Emacs in a way that conforms to macOS security policies. In effect, I made a macOS client app for Emacs server.

This app is named Scrim and is now in BETA test. I’m looking for testers, especially for those of you who already have Captee. If you’re interested and willing to provide feedback, please add yourself to the Beta test group on TestFlight, an Apple tool for distributing code-signed apps. At current I’m most interested in the following feedback:

Sanity check - Does the app work as advertised?
Were the instructions to setup Emacs server easy to follow? If not, then what was missing?
Are you a user of Captee?

If you do sign yourself up for this, please provide answers to the above via the mail feedback in the Scrim app.

Thanks!

YES, I WANT TO BETA TEST SCRIM

Converting a Markdown Region to Org Revisited

2025-01-10T13:00:00-08:00

On more than one occasion I’ve found myself needing to convert Markdown formatted text to Org using this workflow:

Copy Markdown formatted text (usually from a web browser) into the kill ring.
Paste above text into an Org buffer.
Manually reformat said text to Org.

Folks who have pandoc installed can automate step 3 above with the Elisp function quoted below from the Stack Exchange post “Convert region/subtree from Markdown to org”.

(defun my-md-to-org-region (start end)
  "Convert region from markdown to org"
  (interactive "r")
  (shell-command-on-region start end "pandoc -f markdown -t org" t t))

So just copy the above function into your config and call it a day, no?

Not quite. If left unspecified, pandoc will automatically wrap long lines 😞. I prefer keeping a paragraph as a single long line in Org.

To keep a paragraph as a long line, use the pandoc argument --wrap=preserve.

Listed below is an amended version of the above routine.

(defun cc/markdown-to-org-region (start end)
  "Convert Markdown formatted text in region (START, END) to Org.

This command requires that pandoc (man page `pandoc(1)') be
installed."
  (interactive "r")
  (shell-command-on-region
   start end
   "pandoc -f markdown -t org --wrap=preserve" t t))

Regardless of your preference, if this is news to you and seems useful, give it a try.

References

org mode - Convert region/subtree from Markdown to org - Emacs Stack Exchange

Styling Text via Keyboard in Org and Markdown

2024-11-13T17:00:00-08:00

A recent Mastodon post by Christian Tietze asked for how one could style text in Org with a keyboard. There is an existing command org-emphasize that can do this but it has a user interface design flaw: It requires the user to know ahead of time the markup character used to style the text. For many, recalling this Org-specific character is a chore. A friendlier command design would let the user specify the style (e.g. bold, italic, code, etc.) to markup the text.

A while back, I wrote a post describing how to style text in Org and Markdown using mouse-driven menus. Thinking about Tietze’s ask, I realized I had done most of the work towards building a keyboard-driven command to style text. So I set about to followup with making that command. This post details that effort.

In thinking about keyboard-driven text styling, I wanted to achieve two things:

Style text using logical names (e.g. bold, italic, code, …)
Minimize the work required to select the text to be styled.
- Infer from the point what the text region to be styled should be.

A nice feature of the Markdown styling commands is that they will try to infer from the point (cursor position) what region of text to apply the style to. No such facility that I know of exists in Org. But it’s not all gravy for Markdown. The same Markdown styling commands will only work on what Emacs defines to be a word. This makes it a hassle to deal with continuous text (that is, no spaces or linebreaks) that embeds dashes (-) or underscores (_). This is because Emacs navigation by word consider the dash and underscore characters to be word separators.

No worries though as this is Emacs: There’s always a better way.

In building Casual EditKit, I became familiar with commands that operated on balanced expressions (aka sexp). Using them, I observed that they treated continuous alphanumeric text with dashes and underscores as a single entity. With this knowledge I set out to build the command cc/emphasize-dwim.

Here’s a video of it in action.

Closing Thoughts

NGL, I’ve been pleasantly surprised at how well this has worked out, so much so that it’s motivated me to write this post. Interested readers are encouraged to try this code out.

As far as what keybinding to use for cc/emphasize-dwim, I’ve landed on using C-/ for now, which by default is bound to undo. Since my daily driver is macOS, I assign M-z for that job. As always, it’s left to the reader to choose a keybinding of their preference.

Referencing Org Table Cells with Text Regions

2024-10-08T11:00:00-07:00

In a typical spreadsheet app, a common user interface practice is to avoid forcing the user to keep track of cell references as much as possible. This is usually seen in selecting a range of cells to construct a formula, where the user only needs to control the range of selection; the app keeps track of which cell references to pass into the formula.

Sadly, this feature is not available by default in Org tables. But no need to fret; Emacs and Org give you the mechanisms to build a reasonable emulation of spreadsheet-style cell range selection. This post shows you how.

Before going into details, here’s a demo of selecting a row of values and using that selection to compute its sum.

| a | b | c | Total |
|---+---+---+-------|
| 1 | 5 | 2 |       |
| 3 | 4 | 1 |     8 |
| 8 | 1 | 2 |       |
#+TBLFM: $4=vsum(@3$1..@3$3)

In the example above the following steps are taken:

Place the point in the cell to enter a Calc formula, in this case vsum().
- Note that vsum here stands for vector sum.
Select a range of cells, typically with a mouse by pointing and dragging.
- The selection can be either be a region or a region-rectangle.
  - Did you know that C-M-mouse-1 lets you mouse select a rectangle? I didn't until recently 🤯.
Raise a context menu pressing the right mouse button over the selected region.
In the context menu, a menu item showing the selected cell region is displayed (in this case @3$1..@3$3).
1. Select this menu item to copy the cell references into the kill-ring.
Move the point back to the initial cell in step 1 and yank the cell reference as an argument for vsum().
Press TAB or RET to compute the formula.

Some observations:

Throughout this workflow, the user did not need to keep track of cell references.
The resultant table formula #+TBLFM: left-hand-side (lhs) is populated with only the column after step 6.
- The user must amend the lhs to be @3$4 to specify that the formula should only apply to that cell.
- AMENDED 2024-10-10: A workaround for this leveraging org-table-edit-formulas has been made. Details at end of this post.
The table formula can be modified to use relative references at the cost of requiring the user to truly understand Org table reference syntax.
Keyboard enthusiasts can use the command cc/copy-org-table-reference-dwim to convert a text region into an Org table reference.

Intrigued? Okay, let’s do this.

Org Table Reference Background

First off, let’s understand how Org tables address cells. A table cell is addressed using the following representation:

@ROW$COLUMN

A row is prefixed with @ whereas a column is prefixed with $. The row specification is placed before the column specification which is opposite of what's seen in conventional spreadsheets and Cartesian coordinates that specify the column first, then the row. Anyways, water under the bridge. With Org tables, it is always row first, column second.

A reference to an individual cell is called a field reference.

A reference to a set of cells is called a range reference. A range reference is constructed from two field references, where a and c are row values, b and d are column values.

@a$b..@c$d

Range references are translated into a vector of values that can be fed into Calc vector functions. Typically the two field references in a range reference are either in the same row or column. If not, then the vector of values is constructed using the two field references as a bounding box.

Most importantly, a range reference specifies values in a single vector. The orientation (row or column) of a range reference matters. To clarify, this is best illustrated by example.

#+NAME: vector-example
| a | b | c |
|---+---+---|
| 1 | 2 | 3 |
| 4 | 5 | 6 |
| 7 | 8 | 9 |

| Range      | Vector       | Notes                                                  |
|------------+--------------+--------------------------------------------------------|
| @2$1..@2$3 | [1, 2, 3]    | Cell values taken left to right.                       |
| @2$2..@4$2 | [2, 5, 8]    | Cell values "sliced" wrt column specification.         |
| @3$2..@4$3 | [5, 6, 8, 9] | Cell values taken using the extents as a bounding box. |
#+TBLFM: @2$2=remote(vector-example, @2$1..@2$3)
#+TBLFM: @3$2=remote(vector-example, @2$2..@4$2)
#+TBLFM: @4$2=remote(vector-example, @3$2..@4$3)

Design Requirements

With the background out of the way, let’s put down some requirements.

Given an active region within an Org table, translate it to a range reference with the following specifics:
1. If the selected region is a single column or row, return a range reference using the mark and point of the region.
2. If the selected region crosses has multiple rows and columns, then treat the mark and point of the region as the bounding box specification for a matrix represented as a Calc vector of vectors.
3. If the selected region is a in a single cell, only return the field reference for that cell.
4. The bounding box calculation is the same regardless if the mark is before or after the point.
Support both a mouse menu and mini-buffer command interface for obtaining an Org table region reference from a selected text region.

As a table is a convenient representation of a matrix, the above requirements makes the design decision to reflect that in mapping a text region having multiple rows and columns to a vector of vectors instead of a single vector.

Show Code

Here’s code that meets the requirements described above. Commands of note:

cc/copy-org-table-reference-dwim
cc/mouse-copy-org-table-reference-dwim

Given a point or region (text or rectangle) in an Org table, either of the two commands will create an Org table reference and push it in the kill-ring. The first command is intended to be called from the keyboard, the second from a mouse menu.

;; Org Table Cell Reference Functions

(defun cc/org-table-cell-at-point ()
  "At point, return the cell object from an Org table.

A cell object is defined to be a list containing the row and the
column, successively."
  (if (not (org-at-table-p))
      (error "Not in a table"))

  (let* ((row (org-table-current-dline))
         (col (org-table-current-column)))
    (list row col)))

(defun cc/format-org-table-field-reference (cell)
  "Format CELL object into @r$c format.

CELL object obtained via `cc/org-table-cell-at-point'.

See Info node `(org) References' for more on Org table field
reference format."
  (let ((row (nth 0 cell))
        (col (nth 1 cell)))
    (format "@%d$%d" row col)))

(defun cc/org-table-range ()
  "Return range object from a region defined within an Org table.

A range object is a list of two cells computed via
`cc/org-table-cell-at-point', the first being the cell at the
start of the region and the last being the cell at the end of the
region."
  (if (not (and (org-at-table-p) (use-region-p)))
      (error "Not in an Org table"))

  (save-excursion
    (let* ((end (cc/org-table-cell-at-point)))
      (exchange-point-and-mark)
      (let ((start (cc/org-table-cell-at-point)))
        (list start end)))))

(defvar cc/last-org-table-reference nil
  "Last stored Org table reference.

State variable to store an Org table reference (field or range)
to be used in an Org table formula. This variable is set via
`cc/org-table-reference-dwim'

NOTE: This state variable to work-around my lack of clarity on
region and mouse menu interaction.")

(defun cc/org-table-reference-dwim ()
  "Org table reference given point or region is defined.

Return Org table reference (field or range) depending on whether
a point or region is defined in an Org table.

If the region is defined over multiple columns, then a Calc
vector matrix is returned. See Info node `(org) Formula syntax
for Calc' for more.

Calling this function will set `cc/last-org-table-reference'.

See Info node `(org) References' for more on Org table field
reference format."
  (if (not (org-at-table-p))
      (error "Not in an Org table"))

  (cond
   ((use-region-p)

    (let* ((range (cc/org-table-range))
           (start (nth 0 range))
           (end (nth 1 range))
           (msg (format "%s..%s"
                        (cc/format-org-table-field-reference start)
                        (cc/format-org-table-field-reference end))))
      (setq cc/last-org-table-reference (cc/org-table-range-to-reference range))
      msg))

   (t
    (let ((msg (cc/format-org-table-field-reference (cc/org-table-cell-at-point))))
      (setq cc/last-org-table-reference msg)
      msg))))

(defun cc/copy-org-table-reference-dwim ()
  "Copy Org table reference (field or range) into kill ring.

Given a point or region defined in an Org table, add to the
`kill-ring' an Org table field or range reference.

If the region is defined over multiple columns, then a Calc
vector matrix is returned. See Info node `(org) Formula syntax
for Calc' for more.

If the buffer *Edit Formulas* is available (usually via
`org-table-edit-formulas'), the reference will be inserted into
it.

See Info node `(org) References' for more on Org table field
reference format."
  (interactive)
  (if (not (org-at-table-p))
      (error "Not in an Org table"))

  (let ((msg (cc/org-table-reference-dwim))
        (formulas-buffer (get-buffer "*Edit Formulas*")))
    (if formulas-buffer
        (with-current-buffer formulas-buffer
          (insert cc/last-org-table-reference)))
    (message "Range: %s, Copied %s" msg cc/last-org-table-reference)
    (kill-new cc/last-org-table-reference)))

(defun cc/mouse-copy-org-table-reference-dwim ()
  "Copy Org table reference (field or range) into kill ring via mouse.

Given a point or region defined in an Org table, add to the
`kill-ring' an Org table field or range reference.

NOTE: This function is intended to be called from a mouse menu
after `cc/copy-org-table-reference-dwim' is called which will set
`cc/last-org-table-reference'. This is to work-around my lack of
clarity on region and mouse menu interaction.

If the region is defined over multiple columns, then a Calc
vector matrix is returned. See Info node `(org) Formula syntax
for Calc' for more.

If the buffer *Edit Formulas* is available (usually via
`org-table-edit-formulas'), the reference will be inserted into
it. If the point in *Edit Formulas* is at the beginning of line,
it will treat the reference as a left hand side (lhs) assignment.

See Info node `(org) References' for more on Org table field
reference format."
  (interactive)
  (if (not (org-at-table-p))
      (error "Not in an Org table"))

  (when cc/last-org-table-reference
    (let ((msg cc/last-org-table-reference)
          (formulas-buffer (get-buffer "*Edit Formulas*")))
      (if formulas-buffer
        (with-current-buffer formulas-buffer
          (if (bolp)
              (insert (format "%s = " msg))  ; treat reference as lhs assignment
            (insert msg))))
      (message "Copied %s" msg)
      (kill-new msg))))

(defun cc/org-table-range-to-reference (range)
  "Convert RANGE object to Org table reference (field or range).

If the region is defined over multiple columns, then a Calc
vector matrix is returned. See Info node `(org) Formula syntax
for Calc' for more.

See `cc/org-table-range' for more on RANGE object."
  (let* ((start (nth 0 range))
         (end (nth 1 range))
         (a (nth 0 start))
         (b (nth 1 start))
         (c (nth 0 end))
         (d (nth 1 end))

         (r1 (apply #'min (list a c)))
         (c1 (apply #'min (list b d)))

         (r2 (apply #'max (list a c)))
         (c2 (apply #'max (list b d)))

         (rowrange (number-sequence r1 r2))
         (buflist (list)))


    (cond
     ((and (= r1 r2) (= c1 c2))
      (format "@%d$%d" r1 c1 ))

     ((or (= c1 c2) (= r1 r2))
      (format "@%d$%d..@%d$%d" r1 c1 r2 c2))

     (t
      (mapc (lambda (r)
              (push (format "@%d$%d..@%d$%d" r c1 r c2) buflist))
            rowrange)

      (format "vec(%s)"
              (string-join (reverse buflist) ", "))))))

Original Source.

AMENDED 2024-10-10: cc/copy-org-table-reference-dwim and cc/mouse-copy-org-table-reference-dwim modified to insert content into table formula buffer when org-table-edit-formulas is called.

The following abridged code illustrates how one could configure their context menu to show the current Org table reference and copy it into the kill-ring if selected. See Customizing the Emacs Context Menu for more on Emacs context menus.

(defun cc/context-menu-addon-items (menu click)
  "Context menu hook function using MENU and CLICK event.

MENU - menu to be configured.
CLICK - event"
  (save-excursion
    (mouse-set-point click)
    (cc/context-menu-org-table-items menu (not (org-at-table-p)))
    menu))

(defun cc/context-menu-org-table-items (menu &optional inapt)
  "Menu items to populate MENU for Org table section if INAPT nil."
  (when (not inapt)
    (easy-menu-add-item menu nil
                        ["Table Cell Info"
                         cc/mouse-copy-org-table-reference-dwim
                         :label (cc/org-table-reference-dwim)
                         :help "Copy Org table reference (field or range) into kill ring via mouse"])))

(add-hook 'context-menu-functions #'cc/context-menu-addon-items)

Original Source.

All code running on Emacs 29.4.

Examples

The following table illustrates the result of selecting the text region between the cell with value 2 (@2$2) and the cell with value 12 (@4$4) and inserting its translated Org table reference into different formulas.

The first formula in @2$5 is just the matrix representation between 2 and 12.

The second formula takes the transpose (trn) of the first formula.

The third formula uses some matrix arithmetic to get the sum of each row of the matrix in @2$5.

#+NAME: matrix-example
| a |  b |  c |  d | result                               |
|---+----+----+----+--------------------------------------|
| 1 |  2 |  3 |  4 | [[2, 3, 4], [6, 7, 8], [10, 11, 12]] |
| 5 |  6 |  7 |  8 | [[2, 6, 10], [3, 7, 11], [4, 8, 12]] |
| 9 | 10 | 11 | 12 | [9, 21, 33]                          |
#+TBLFM: @2$5=vec(@2$2..@2$4, @3$2..@3$4, @4$2..@4$4)
#+TBLFM: @3$5=trn(vec(@2$2..@2$4, @3$2..@3$4, @4$2..@4$4))
#+TBLFM: @4$5=@2$5 * [1 1 1]

Closing Thoughts

Manually dealing with Org table references is both tedious and error-prone. Automating their generation using text range selection seems like a good way to make this more bearable. If you're comfortable with Elisp, I recommend you give this code a try.

I’m considering publishing this code as a package on MELPA but I think the utility of it is high enough to merit being in Org itself. For next steps, I look forward to finding out if that is true.

Amendment 2024-10-10

In sharing this post at the 2024-10-09 Org Meetup, I was made aware by Ihor Radchenko that the point position in the special buffer (named *Edit Formulas*) created by the org-table-edit-formulas command (binding C-c ') will highlight any field or range reference. This special buffer provides context from which we can enhance the commands cc/copy-org-table-reference-dwim and cc/mouse-copy-org-table-reference-dwim to insert their results directly into it, removing the step to yank. cc/mouse-copy-org-table-reference-dwim is further enhanced to check if the point in *Edit Formulas* is also at the beginning of a line. If so, then the inserted field reference will be treated as the lhs argument of a formula. This approach can be viewed as a workaround the direct entry of a formula in a cell with the "=" character, which will only insert in the lhs a column reference.

Announcing Casual Agenda

2024-09-03T11:00:00-07:00

Use Org mode long enough and invariably you'll get around to using Agenda views to manage your defined tasks. Agenda views have become core to my Emacs experience and I'm certain that I'm not alone in saying this. As with other packages, there's a steep learning curve to using Agenda views, largely because Org adopts the Emacs convention of using keybindings and the mini-buffer prompt as its primary user interface with an added twist of frequently making this behavior conditional on where the point is.

To help ease using Agenda views, I'm happy to announce Casual Agenda, a Transient user interface for Org Agenda available on MELPA today.

Casual Agenda organizes the Agenda view commands into the following sections:

Agenda - Modify the view duration (day, week, month, year).
Filter - Filter displayed headlines with different criteria.
Actions - Perform an activity on a headline, create/capture a headline, or even generate a different agenda view.
Navigation - move the point to where you want it to be.
Utils - Set a timer, get almanac info.

Note that all the menus in Casual Agenda are contextual so as one navigates the Agenda view, the menu items will be enabled as appropriate.

Operating on Headlines

Use the Operations menu to alter attributes about it such as TODO state, scheduling, tags, and priority. To use it, move the point to the line of a heading you wish to change and from the main menu select “(o) Operations”. The following menu will be displayed.

Marking Headlines

User the Mark menu to mark different headlines and perform a bulk action on them. From the main menu, select “(m) Mark” to display the following menu:

Changing Modes and Settings

Agenda views have different display modes and behavior that can be modified from the Settings menu. From the main menu, select “(,) Settings” to display the following menu:

Almanac

Get sunrise/sunset times, lunar cycle dates, and holidays with respect to a date via the Almanac menu. From the main menu, select “(l) Almanac” to display the following menu.

Unicode Symbol Support

By enabling “(u) Use Unicode Symbols” from the Settings menu, Casual Agenda will use Unicode symbols as appropriate in its menus.

Installation Notes

Casual Agenda requires that you have a relatively recent version of Org (≥ 9.7.1) and Transient, both built-in packages to Emacs 29.1+. To update those packages you will need to take into consideration Emacs' default policy of not upgrading built-in packages unless configuring it to explicitly do so. My last blog post provides more detail on this.

Closing Thoughts

As with the other Casual Suite packages, Casual Agenda is intended to lower the cognitive load of using Agenda views. Even as I’ve already committed to memory many of these commands, it’s often that I mess up by issuing the wrong binding at the wrong place or just plain failing to recall a command. Since building and using Casual Agenda, this has happened less so. You might perhaps share the same experience.

Exporting UTF-8 Smart Quotes from Org Mode

2024-04-10T14:30:00-07:00

Count me as one who really likes using smart quotes when exporting from Org. To enable this requires that you set the variable org-export-with-smart-quotes to t. So far, so good yes? Not quite. Apparently by default when exporting to HTML or Markdown, Org will use the HTML character entity representation of a smart quote which I see as overly conservative in 2024. As most contemporary software now can handle UTF-8, it seems to me more sensible to use smart quote characters that are likewise UTF-8. For English these are:

left and right double quote
left and right single quote
apostrophe (right single quote)

Apparently Org has an alist variable that captures the different representations of the above for different languages and for different export targets. This variable is called org-export-smart-quotes-alist and sure enough it captures the UTF-8 set of characters for each language. Empowered with this knowledge, here’s some Elisp code to reconfigure org-export-smart-quotes-alist so that exporting to HTML or Markdown uses the UTF-8 representation for English (en).

(defun cc/reconfig-org-smart-quotes-lang (lang)
  "Reconfigure Org smart quotes to use utf-8 per LANG."
  (let* ((db-entry (assoc-string lang org-export-smart-quotes-alist))
         (utf8-primary-opening (plist-get (assoc-default 'primary-opening db-entry) :utf-8))
         (utf8-primary-closing (plist-get (assoc-default 'primary-closing db-entry) :utf-8))
         (utf8-secondary-opening (plist-get (assoc-default 'secondary-opening db-entry) :utf-8))
         (utf8-secondary-closing (plist-get (assoc-default 'secondary-closing db-entry) :utf-8))
         (utf8-apostrophe (plist-get (assoc-default 'apostrophe db-entry) :utf-8))
         )

    (setf (plist-get
           (assoc-default 'primary-opening
                          (assoc-string lang org-export-smart-quotes-alist))
           :html)
          utf8-primary-opening)

    (setf (plist-get
           (assoc-default 'primary-closing
                          (assoc-string lang org-export-smart-quotes-alist))
           :html)
          utf8-primary-closing)

    (setf (plist-get
           (assoc-default 'secondary-opening
                          (assoc-string lang org-export-smart-quotes-alist))
           :html)
          utf8-secondary-opening)

    (setf (plist-get
           (assoc-default 'secondary-closing
                          (assoc-string lang org-export-smart-quotes-alist))
           :html)
          utf8-secondary-closing)

    (setf (plist-get
           (assoc-default 'apostrophe
                          (assoc-string lang org-export-smart-quotes-alist))
           :html)
          utf8-apostrophe)))

(add-hook 'org-mode-hook (lambda ()
                           (cc/reconfig-org-smart-quotes-lang "en")))

The above code is tested on Emacs 29.1, Org 9.6.25.

If you export to different languages, be sure to run cc/reconfig-org-smart-quotes-lang with your language of choice.

I leave it as an exercise to motivated readers more experienced in Elisp to write a cleaner version of this code. As it stands, this works well enough to achieve my desired goal.

Computing Truth Tables in Org

2024-02-15T10:30:00-08:00

Truth tables are great way to design and verify combinational logic. Useful as truth tables are, they however can be onerous to write: as the number of inputs grow, so does the size of table in exponential fashion (row size = 2^{number of inputs}). Writing this out is not fun.

Thankfully, enumeration of said input values in Emacs can be automated. This post goes over an approach to generating truth table inputs with which you can use to drive combinational logic in Org.

Inserting the Truth Table Input

Before getting into implementation details, let’s go over inserting truth table input into your current Org buffer.

Invoke the command M-x cc/insert-truth-table-input. In the mini-buffer you will be prompted for the number of inputs to enumerate for the truth table input values. Entering the value 2 will insert the following table into your buffer. Note that the second and third columns map to 2¹ and 2⁰ respectively. Reading across a row, the first column shows the decimal value of the input, whereas the remaining columns shows the binary representation of that input value.

|i|1|0|
|-+-+-|
|0|0|0|
|1|0|1|
|2|1|0|
|3|1|1|

Combinational Logic in Org

If the truth table input is in an Org buffer, we can use its spreadsheet capabilities to compute combinational logic with said values. For example, with the table above we can compute the xor of each input row by entering the following formula for all cells in column $4. Refer to Emacs Lisp forms as formulas for more general documentation on how Org formulas work.

| i | 1 | 0 | output |
|---+---+---+--------|
| 0 | 0 | 0 |      0 |
| 1 | 0 | 1 |      1 |
| 2 | 1 | 0 |      1 |
| 3 | 1 | 1 |      0 |
#+TBLFM: $4='(digital-xor $2 $3);N

Note that the formula must include the N flag so that referenced cells are parsed as numbers.

To support combinational logic in Org tables, I’ve created a library to support digital boolean operations, where the only legal input values are 0 and 1. The following boolean logic operations are supported:

multi-input and (digital-and)
multi-input or (digital-or)
multi-input nand (digital-nand)
multi-input nor (digital-nor)
2-input xor (digital-xor)

Implementation

Implementation of the behavior described above is done in the following two files:

cc-truth-table.el which holds the functions required to implement cc/insert-truth-table-input.
cc-digital-logic.el which holds all the functions required to support digital combinational logic with tables created by the above.

All code verified with Emacs 29.1.

Installation

Install the above files into your load-path. Then include the following lines in your Emacs initialization file.

(require 'cc-truth-table)
(require 'cc-digital-logic)

As this work is relatively new, I’ve yet to try to make this into a MELPA package for public distribution. If enough interest is there, I can be motivated to do this.

On the need to write a digital logic library

At the start I tried to get away with using the base Elisp functions for boolean operations. I quickly found out that such functions were really designed presuming the input values are t or nil. Using the digital convention of using 0 and 1 as input values to the base Elisp boolean operations can lead to surprising results. For example, evaluating (not 0) will return nil. Ultimately this led to writing a library of boolean functions expressly designed to use as input the digital values of 0 and 1.

Closing Thoughts

Programmers deal with combinational logic all the time. As much as I love truth tables, they are a PITA to make because so much of the work is mechanical. Automating this work with Elisp and Org has freed me to focus on just solving the logic problem. Engaged readers might find similar results.

Thanks for reading! Feel free to provide feedback about this post to me on Mastodon.

Running SQL Queries on Org Tables

2023-11-27T15:00:00-08:00

One of the best things in Org Mode are tables and if you know SQL, you have at your disposal the ability to process your tables like a SQL database. Even better, you can run a SQL query on your table without the overhead of manually creating the database. This post shows you how.

SQL Query on Org Table Workflow

To implement an SQL query on an Org table we do the following:

Convert the Org table to a SQLite table via the Org Babel SQLite integration.
- Inspect the Org table columns to determine its SQL type.
- Auto-generate the SQL CREATE TABLE statement using the above column types.
- Store the table in an in-memory database.
Run a SQL query on the above created SQLite table via Org Babel and render its result as an Org table.

Requirements

All code here has been tested on Emacs 29.1 and Org 9.6.11.

Have Org Babel SQLite support installed. Instructions for doing this are provided on the Org website.

Demo

Given the following named table example_table, find the aggregate total units per city.

#+NAME: example_table
| site | city          | units |
|------+---------------+-------|
|    1 | San Francisco |   182 |
|    2 | San Francisco |    82 |
|    3 | San Francisco |   124 |
|    4 | Berlin        |   105 |
|    5 | Berlin        |    92 |
|    6 | Seoul         |   104 |
|    7 | Seoul         |    84 |
|    8 | Seoul         |    97 |
|    9 | New York      |   103 |
|   10 | New York      |   140 |

To convert the above Org table to a SQLite table, we will call the named code block cc/org-table-to-sqlite with two arguments as shown below:

table which is a reference to the named table, and
table-name which is the string representation of table above.

The table name must be SQL-legal.

#+CALL: cc/org-table-to-sqlite(table=example_table, table-name="example_table")

Invoking C-c C-c (org-ctrl-c-ctrl-c) on the above will output the following generated Org SQLite code block.

#+RESULTS:
#+begin_src sqlite :db ":memory:" :var orgtable=example_table :colnames yes
drop table if exists example_table;
create table example_table(site INT, city TEXT, units INT);
.import $orgtable example_table

-- Edit SQL Query Below
select site, city, units from example_table;
#+end_src

A couple of observations of the above generated code:

The database is in-memory allowing for flexibility in changing the Org table.
Every time the code block is executed, the SQL table is dropped if it already exists.
The SQLite column types are determined by examining each column element. (Note: implementation is O(n))
An example SQL SELECT statement is generated using the Org table header.

As we want to determine the aggregate total units per city, let’s modify the above SELECT statement as follows:

select site, city, sum(units) as totals from example_table group by city order by totals;

Invoking C-c C-c on the generated and edited code block yields the following result:

#+RESULTS:
| site | city          | totals |
|------+---------------+--------|
|    4 | Berlin        |    197 |
|    9 | New York      |    243 |
|    6 | Seoul         |    285 |
|    1 | San Francisco |    388 |

Implementation

The Org Elisp code block cc/org-table-to-sqlite is stored in the file cc-org-table-to-sql.org and is posted as a GitHub gist (note click on “Raw” to see all Org markup).

It can be loaded via the org-babel-lob-ingest command as follows:

1	`(org-babel-lob-ingest "cc-org-table-to-sql.org")`

Remembering how to invoke the code block cc/org-table-to-sqlite can be bit much. Thankfully we can use YASnippet to help. Shown below is a snippet for creating a named table and call to the code block cc/org-table-to-sqlite referencing said named table.

# name: org-table-to-sqlite
# key: otsql_
# --
#+NAME: ${1:example_table}
| ${2:a} | ${3:b} |
|---|---|
|  1| 10|
$0
#+CALL: cc/org-table-to-sqlite(table=$1, table-name="$1")

Closing Thoughts

Turning an Org table into an ad-hoc SQL database opens a world of possibilities for data journaling and analysis. This post merely scratches the surface of what can be done.