liam's blog

Python-in-the-browser

Fri, 13 Dec 2024 05:00:00 +0000

I recently wrote and distributed a Python package for a project, and I wanted a web interface to go with it. Naturally, I chose React to build it, but I had never called Python from a React app before, so I had to figure things out.

After a quick search, I stumbled across Pyodide, which is a port of CPython to WebAssembly/Emscripten. It allows users to install and use any pure Python package with a wheel on PyPI via micropip. This sounded like something that could be useful, so I decided to try it out.

Since my package was already distributed, all I needed to do, according to the Pyodide documentation, was:

npm install pyodide
Install my package with micropip
Run arbitrary Python code using my package with Pyodide’s API

Turns out this worked for my use case, let’s dive deeper into each step!

I. Installation

This is straightforward. I used bun as my package manager, so all I had to do was run bun add pyodide. Other package managers should follow a similar command. You can check out the official Pyodide package on the npm registry.

II. Loading the package

Getting Pyodide up and running involves a few steps that need to happen in the right order. First, we need to load the core Pyodide runtime from a CDN. Then we can use Pyodide’s package manager, micropip, to install our Python package. This is similar to how you might use pip in a regular Python environment, but it’s all happening in the browser.

One nice thing about Pyodide’s CDN distribution is that it includes many common Python packages pre-built and ready to use. However, for packages that aren’t included by default (like my pqg package), we need to install them explicitly using micropip. Here’s the complete initialization sequence:

import { PyodideInterface, loadPyodide } from 'pyodide';

// Initialize Pyodide with the specified CDN URL
// This loads the core Python runtime environment into the browser
const client = await loadPyodide({
  indexURL: 'https://cdn.jsdelivr.net/pyodide/v0.26.3/full',
});

// Load the micropip package, which is Pyodide's package installer
// This is similar to pip for regular Python, but works in the browser
await pyodide.loadPackage('micropip');

// Import the micropip module into the JavaScript environment
// This allows us to use micropip's functions from JavaScript
const micropip = pyodide.pyimport('micropip');

// Use micropip to install the 'pqg' package from PyPI
// This downloads and installs the pure Python package in the browser environment
await micropip.install('pqg');

It’s worth noting that this initialization process can take a few seconds, especially on slower connections. The Pyodide runtime itself is about 6MB, and any additional packages you install will add to that. In my case, I found that showing a loading indicator during this initialization was essential for a good user experience.

Also, because these operations are asynchronous, you need to be careful about when you start trying to execute Python code. I learned the hard way that trying to run code before Pyodide is fully initialized will fail. This is one of the reasons I eventually moved all of this into a web worker, which we’ll discuss later.

III. Running code

Now that we have Pyodide and our package loaded, we need to actually run some Python code. In my case, I needed to generate pandas queries based on user-defined schemas and settings. One early challenge I encountered was that Pyodide doesn’t support multiprocessing due to browser limitations - I had to explicitly disable it in my package’s configuration to ensure compatibility. Here’s how I structured the code generation (abridged):

// Generate Python code that creates and configures the query generator
const generateQueryGenerationCode = (schema: string, settings: Settings) => `
  import json

  from pqg import Generator, QueryStructure, Schema, QueryPool, GenerateOptions

  # Parse the schema from JSON
  schema = Schema.from_dict(json.loads('''${schema}'''))

  # Configure query structure based on user settings
  query_structure = QueryStructure(
    groupby_aggregation_probability=${settings.groupbyProbability},
    max_groupby_columns=${settings.maxGroupbyColumns},
    # ... other settings
  )

  generator = Generator(schema, query_structure)

  generate_options = GenerateOptions(
    ensure_non_empty=${settings.enforceNonEmptyResults ? 'True' : 'False'},
    multi_line=${settings.multiLine ? 'True' : 'False'},
    multi_processing=False,
    num_queries=${settings.numQueries}
  )

  query_pool = generator.generate(generate_options)

  # Return results as JSON
  json.dumps({
    'queries': [str(query) for query in query_pool],
    # ... other keys
  })
`;

// Execute the code using Pyodide
const response = await client.runPythonAsync(generateQueryGenerationCode(schema, settings));

The key here is that we’re generating a Python code string that will be executed by Pyodide. We interpolate JavaScript values (the schema and settings) into the Python code string. This allows us to pass configuration from our React UI to the Python runtime.

One important detail is that we need to handle the data serialization carefully. Since we’re crossing the JavaScript-Python boundary, we use JSON as our data format. The Python code parses the JSON schema and returns its results as JSON, which can then be parsed back into JavaScript objects.

IV. Inside a web worker

When I first implemented this, I quickly realized that loading and running Python in the main thread wasn’t ideal - it would freeze the UI during initialization and execution. The solution was to move all Pyodide-related code into a web worker.

Here’s how I built the worker interface (abridged):

import { useEffect, useRef, useState } from 'react';

import { useToast } from './use-toast';

type WorkerStatus = 'idle' | 'loading' | 'ready' | 'error';

type WorkerResponse =
  | { type: 'status'; payload: WorkerStatus }
  | { type: 'error'; payload: string }
  | { type: 'result'; payload: string };

export const usePyodideWorker = () => {
  const [status, setStatus] = useState<WorkerStatus>('idle');

  const workerRef = useRef<Worker>();

  const { toast } = useToast();

  useEffect(() => {
    workerRef.current = new Worker(
      new URL('../workers/pyodide.worker.ts', import.meta.url),
      { type: 'module' }
    );

    workerRef.current.onmessage = (event: MessageEvent<WorkerResponse>) => {
      const { type, payload } = event.data;

      switch (type) {
        case 'status':
          setStatus(payload);
          break;
        case 'error':
          // ...
          setStatus('error');
          break;
      }
    };

    workerRef.current.postMessage({ type: 'init' });

    return () => {
      workerRef.current?.terminate();
    };
  }, [toast]);

  const runPython = async <T>(code: string): Promise<T> => {
    return new Promise((resolve, reject) => {
      // ...
    });
  };

  return {
    error: status === 'error',
    loading: status === 'loading',
    ready: status === 'ready',
    runPython,
  };
};

I created this as a React hook that manages the worker lifecycle. The hook creates the worker when the component mounts and provides a runPython function that components can use to execute Python code. The status updates help the UI show loading states while Pyodide initializes or executes code.

The worker implementation itself is relatively straightforward:

import { PyodideInterface, loadPyodide } from 'pyodide';

let pyodide: PyodideInterface;

type WorkerMessage =
  | { type: 'init' }
  | { type: 'run'; payload: { code: string } };

type WorkerResponse =
  | { type: 'status'; payload: 'loading' | 'ready' }
  | { type: 'error'; payload: string }
  | { type: 'result'; payload: string };

self.onmessage = async (event: MessageEvent<WorkerMessage>) => {
  const { type } = event.data;

  try {
    switch (type) {
      case 'init':
        if (!pyodide) {
          self.postMessage({
            type: 'status',
            payload: 'loading',
          } satisfies WorkerResponse);

          pyodide = await loadPyodide({
            indexURL: 'https://cdn.jsdelivr.net/pyodide/v0.26.3/full',
          });

          await pyodide.loadPackage('micropip');

          const micropip = pyodide.pyimport('micropip');
          await micropip.install('pqg');

          self.postMessage({
            type: 'status',
            payload: 'ready',
          } satisfies WorkerResponse);
        }
        break;
      case 'run':
        if (!pyodide) throw new Error('Pyodide not initialized');

        self.postMessage({
          type: 'result',
          payload: await pyodide.runPythonAsync(event.data.payload.code),
        } satisfies WorkerResponse);

        break;
      default:
        throw new Error(`Unknown message type: ${type satisfies never}`);
    }
  } catch (error) {
    self.postMessage({
      type: 'error',
      payload: error instanceof Error ? error.message : 'Unknown error',
    } satisfies WorkerResponse);
  }
};

The worker maintains the Pyodide instance and handles two types of messages: initialization and code execution. This keeps all the heavy lifting off the main thread, ensuring the UI stays responsive.

One thing to note is that web workers have their own global scope, so the Pyodide instance and any installed packages persist between executions. This is actually beneficial as it means we only need to load Pyodide and install packages once, rather than for each query generation.

Fin

Building this interface turned out to be surprisingly straightforward thanks to Pyodide. The ability to run Python code directly in the browser, without needing a backend server, makes for a great developer experience. The combination of React for the UI, Pyodide for Python execution, and web workers for performance created a smooth, interactive interface for my Python package.

While there are some limitations - Pyodide can only run pure Python packages, and the initial load time can be significant - it’s a powerful tool for the right use case. In my case, it allowed me to create a web interface for my Python package with minimal overhead and deployment complexity.

The full code for the implementation can be found on GitHub.

Incremental-select-with-treesitter

Sat, 09 Jul 2022 04:00:00 +0000

treesitter is a tool that can build a concrete syntax tree for a source file and incrementally update the tree as the file is modified.

Recently, I’ve been experimenting more and more with treesitter’s query DSL and an abstraction layer in the form of a neovim plugin, and found a configuration option to be particularly useful when editing code: incremental selection.

When you write a treesitter parser, you must write a grammar that treesitter can read, which it then uses to generate the parser. This grammar specifies the named syntax nodes in the tree and how they relate to other nodes.

For instance, if we head over to the syntax tree playground and parse rust source code that looks like this:

fn main() {
  println!("Hello, world!");
}

we get the following syntax tree output:

source_file [0, 0] - [3, 0]
  function_item [0, 0] - [2, 1]
    name: identifier [0, 3] - [0, 7]
    parameters: parameters [0, 7] - [0, 9]
    body: block [0, 10] - [2, 1]
      macro_invocation [1, 2] - [1, 27]
        macro: identifier [1, 2] - [1, 9]
        token_tree [1, 10] - [1, 27]
          string_literal [1, 11] - [1, 26]

The neovim plugin written as an abstraction layer for treesitter usage within neovim is able to parse your source file as you type. That is, it’s able to compute a syntax tree that resembles the one shown above, on the fly.

As you hover over a certain part of a file, it is able to detect which node you’re currently at in the tree.

The plugin includes a configuration option which allows for the setting of keybindings for initializing a node selection and incrementing a selected node range – configured like so:

incremental_selection = {
  enable = true,
  keymaps = {
    init_selection = "<cr>",
    node_incremental = "grn",
    scope_incremental = "grc",
    node_decremental = "grm",
  }
}

Incremental selection has improved the rapidity in which I can yank or replace blocks of text in treesitter parsed source files, as expanding a node selection with a keybinding is super fast and usually results in the exact block of text I want to modify.

Here’s a quick demo of it in action:

A reference of my full neovim configuration can be found on github, in my dotfiles repo. Be sure to check it out!

Powerful-`just`-features

Tue, 12 Apr 2022 04:00:00 +0000

just is a command runner written in Rust. Commands, or recipes in just lingo, are populated in a file called ‘justfile’, and are expressed in a make-like syntax.

I’ve been using just as my command runner for all of my projects, big or small, and find it extremely useful to have a file in the projects root that contains every command needed to interact with the project.

There are quite a few features baked into this program that make it unique from similar projects, which I find nice to have when crafting software. This post simply documents the few I use most often.

Aliases
Fuzzy searching recipes
Recipes in arbitrary languages
Taking advantage of the default recipe
Recipe dependencies
Format your justfile
Documenting recipes

Aliases

Recipes can have aliases, or alternative names which refer to the original recipe that can be used with just to invoke the original recipe.

For instance, if you have a recipe called run, you can alias it to r like so:

alias r := run

run:
  cargo run

and invoke it with just r.

Somewhat related side note: consider aliasing the just command to j, it’ll save you some typing if you frequently execute commands with just.

Fuzzy searching recipes

One feature I discovered recently is just --choose, which lets you fuzzy search through recipes defined in your justfile. You can also specify a chooser by passing it into just --chooser, else it will default to using fzf.

There is work planned to support recipe previews and command line arguments for recipes that take them when using the default chooser.

However, with the help of just --summary, just --show and a bit of Python, here’s how you can get to look:

Recipes in arbitrary languages

just recipes can be expressed in another language by adding a shebang at the top of the recipe, as they get executed as scripts by the shell.

For instance, we can write a small script in Python that lists the contents of the current directory, and put it in a recipe:

ls:
  #!/usr/bin/env python3
  import os
  print(*os.listdir())

Taking advantage of the default recipe

just allows you to specify a default recipe that gets executed when invoking just without any arguments in a directory that contains a justfile (or a sub-directory in which a parent directory contains a justfile).

For instance:

default:
  just --list

I usually use this feature to list out all recipes that are available in the justfile however this could be used for anything that merits being quickly invoked with a single command.

Recipe dependencies

just recipes can contain ‘dependencies’, or other recipes that should be ran before a given recipe.

For example, let’s say you have a ‘build’ recipe and want to run tests – specified in a ‘test’ recipe – each time before executing the contents of the recipe, you can do so by making ‘test’ a dependency of ‘build’, like so:

default:
  just --list

build: test # this is a dependency
  cargo build

test:
  cargo test

This also comes in handy when you want to run multiple recipes in sequence, just allows for a very expressive syntax via dependencies:

default:
  just --list

all: build test fmt

build:
  cargo build

test:
  cargo test

fmt:
  cargo +nightly fmt

Format your justfile

As of recently, just supports justfile formatting via the --fmt command.

It currently requires the --unstable flag, like so:

$ just --fmt --unstable

Invoking the command above can turn a justfile that looks like this:

default:
    just --list

build:
        cargo build

test:
    cargo test

clippy:
            cargo clippy --all-targets --all-features

into this:

default:
  just --list

build:
  cargo build

test:
  cargo test

clippy:
  cargo clippy --all-targets --all-features

This can be useful if you have many people working on your project and recipes get added or modified in your projects justfile on the regular.

Documenting recipes

just allows for comments above recipes that get appeared when invoking just --list.

For example, here is a sample justfile with a run recipe that contains a comment above it:

# Run the project
run:
  cargo run

invoking just --list will yield the following output:

Available recipes:
    run # Run the project

This is extremely useful for documenting obscure commands, especially when working in teams where people come and go on a project frequently.

Fin

I personally think any project could benefit greatly by having a single source of truth in regards to project specific commands, and just provides a simple way to set that up.

Have a look over at the official readme document on GitHub for more information and further elaboration on the features mentioned here.

Scripting-in-markdown

Sun, 27 Feb 2022 05:00:00 +0000

I recently wrote yet another command-line utility in Rust called present that lets you interpolate the standard output of arbitrary commands that get interpreted by the shell into a markdown file, and I thought I’d share a bit about it here.

Problem

The main problem I kept running into that pushed me to write this tool was having to manually update the outputs of command line utility help messages in my project readme documents.

For instance, I wrote a command-line utility called vim-profiler that lets you profile the startup-time for your installed vim plugins and receive a nicely formatted output. In the readme document I have a usage section that includes the output of calling vp --help. Each time I make a version change, update the description, or update the API – the result of calling vp --help changes, which prompts me to update the readme.

Solution

Instead of invoking the binary with the --help flag, piping the result to pbcopy and then manually pasting the chunk into the appropriate section within the readme myself, I can now have present do all of that for me.

Here’s how that looks like:

Include the command at the start of a fenced codeblock using the present prefix

## vim-profiler 🕒

...

### Usage

```present cargo run -- --help
```
...

Run present on the markdown file

$ present --in-place --path readme.md

View the modified document!

## vim-profiler 🕒

...

### Usage

```cargo run -- --help
vim-profiler 0.0.4
A vim profiling tool.

USAGE:
    vp [FLAGS] [OPTIONS]

FLAGS:
    -h, --help       Prints help information
    -r, --reverse    Display the plugin times in reverse order (fastest first)
    -s, --sys        Show system plugins in the output
    -V, --version    Prints version information
    -v, --verbose    Add informative messages during program execution

OPTIONS:
    -c, --command   <command>      The command to run, e.g vim or neovim [default: vim]
    -n, --count     <count>        The number of plugins to list in the output
    -e, --export    <path>         Export the results to a CSV file
    -f, --file      <file>         A file to open
    -i, --iter      <iter>         The number of iterations
    -p, --plot      <path>         Plot the data and save it to a SVG file
    -x, --precision <precision>    Precision in the output
```
...

In practice, I’ll add the command in a justfile all recipe that I invoke before I commit any changes, such as shown in this justfile.

Moving forward

The project is still in its early stages of development, and there most certainly exists a few non-trivial bugs to be found and fixed.

Some things on the todo list, in no particular order, include:

Improve diff output in interactive mode

The project uses the similar crate to help with diff output in interactive mode, but it can be made nicer by taking advantage of additional features the crate has to offer.

Support same-line command interpolation

This isn’t that important, but it would be nice to support having backticks remain on the same line and have the command result get interpolated with the appropriate newlines. e.g

-> ```echo foo```

-> ```echo foo
   foo
   ```

Handle quotes as a single argument

This would let you actually write inline bash scripts, e.g /bin/bash -c 'for i in *; do echo "$i"; done'. As of right now, the program just splits the entire command string on whitespace, which the shell doesn’t like in certain situations.

This however, for now, can be hacked around by simply including the script in a justfile or makefile and invoking just <name> or make <name> within the markdown file.

Get rid of pulldown_cmark

The project uses the pulldown_cmark crate to get full codeblock ranges within the source, and then hackily turns them into two separate ranges (start, end) which represent the starting and ending range of a single codeblock. This could probably be done better, perhaps without depending on a library for that initial step.

Overall, I get what I need out of the program as it is. Whether or not the aforementioned todo’s provide useful to someone else will most likely have a non-trivial amount of impact on my motivation for getting them done.

Feel free to check out the code over on github https://github.com/terror/present. It is licensed under the highly permissive CC0-1.0 license.

From-bash-to-vim-and-back

Fri, 25 Feb 2022 05:00:00 +0000

I recently ran the following command to get my top 5 most used Bash commands:

$ history | awk '{print $2}' | sort | uniq -c | sort -nr | head -5

and here is what came up:

20793 lt
10124 lv
9328 v
8686 j
7243 gst

These are, for the most part – ordinary, everyday commands:

lt: alias for tree -L 1
v: alias for nvim
j: alias for just, the fabulous command runner
gst: alias for git status

What I assume isn’t so obvious is lv, which stands for ‘last vim’. lv is an alias for the command nvim -c "normal '0" -c bd1, which opens the last opened file at the last cursor position in a new Neovim buffer.

To break the command down a bit:

nvim: the Neovim binary
-c <cmd>: Execute <cmd> after config and first file
normal '0: The last set cursor position and file
bd1: Delete the first buffer

I use Neovim alot, and sometimes I quit the editor to run some arbitrary Bash command. Without having to search for the previously opened file, I invoke lv, which gets me back to where I was at.

From bash to vim and back, in a flash.

Ergonomic-error-handling-in-Rust

Sun, 22 Aug 2021 04:00:00 +0000

There are many ways you can handle errors in Rust programs: panic at every fallible situation, propagate foreign errors up the call stack, craft your own error type.

I’ve recently fallen into a comfortable pattern when dealing with errors in Rust and believe it serves great for small or large projects.

This post will walk through a few different ways to handle errors and then ultimately introduce the snafu library as a clean and ergonomic solution to the elusive task of dealing with erroneous behaviour.

Panic! at the disco
Recoverable errors with Result<T, E>
Crafting custom error types
Enter snafu

Panic! at the disco

Let’s say we have some arbitrary text file on disk that we want to read from, and depending on the content of the file, we want to write back some bytes to it. Simple enough?

use std::{
  fs::{self, OpenOptions},
  io::prelude::*
};

const PATH: &str = "file.txt";

fn run() {
  let content = fs::read_to_string(PATH).unwrap();

  let mut file = OpenOptions::new()
    .write(true)
    .append(true)
    .open(PATH)
    .unwrap();

  if content.trim() == "hello" {
    file.write_all(b"world!").unwrap();
  }

  Ok(())
}

fn main() {
  run();
}

This program can fail in a few different ways:

The file doesn’t exist.
The contents of the file aren’t valid UTF-8.
The call to write_all gets interrupted.

We are handling each of these instances with our calls to unwrap on the fallible function calls. However, instead of being able to recover from an error if it occurs, the current thread panics, which, if it is the main thread, terminates all threads and ends the program with code 101.

Recoverable errors with Result<T, E>

Instead of calls to panic! each time the program can fail, we should be writing more robust code by giving our caller a chance to recover from this behaviour.

use std::{
  fs::File,
  io::{self, prelude::*},
};

const PATH: &str = "file.txt";

fn run() -> Result<(), io::Error> {
  let content = fs::read_to_string(PATH)?;

  let mut file = OpenOptions::new()
    .write(true)
    .append(true)
    .open(PATH)?;

  if content.trim() == "hello" {
    file.write_all(b"world!")?;
  }

  Ok(())
}

fn main() {
  match run() {
    Ok(()) => {},
    Err(e) => eprintln!("{}", e)
  }
}

The Result type is an enum which has an Ok and an Err variant. The Ok variant holds a generic value T that gets returned in the case that no errors have occurred. The Err variant holds a generic value E, usually an error type, if an error has occurred.

The ? operator only works when the function we’re applying it to returns a Result in which whose Err variant contains the error type we wish to propagate. It is essentially a shorthand for the following code:

match result {
  Ok(v)  => v,
  Err(e) => Err(e)
}

This implementation allows for our caller to recover from any erroneous behaviour. However, we can only return one type of error from our function. This prompts anyone who wishes to possibly return more than one error type to create their own type that contains multiple variants.

Crafting custom error types

We can craft our own custom error type as a Rust enum and add multiple variants to it. We can then define a Result type which we can use throughout our program which contains our custom error type as the error to propagate.

In order to demonstrate the need for multiple error types, we’ll extend the functionality of this tiny application to include a call to some API endpoint that is written in the text file, using the reqwest library.

use std::{
  fmt::{self, Display},
  fs::{self, OpenOptions},
  io::{self, prelude::*},
};
use reqwest::blocking;

type Result<T, E = Error> = std::result::Result<T, E>;

#[derive(Debug)]
enum Error {
  IoError(io::Error),
  ApiError(reqwest::Error)
}

impl From<io::Error> for Error {
  fn from(err: io::Error) -> Error {
    Error::IoError(err)
  }
}

impl From<reqwest::Error> for Error {
  fn from(err: reqwest::Error) -> Error {
    Error::ApiError(err)
  }
}

impl Display for Error {
  fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
    match self {
      Error::IoError(e) => write!(f, "IO Error: {}", e),
      Error::ApiError(e) => write!(f, "Api Error: {}", e)
    }
  }
}

const PATH: &str = "file.txt";

fn run() -> Result<()> {
  let content = fs::read_to_string(PATH)?;

  let mut file = OpenOptions::new()
    .write(true)
    .append(true)
    .open(PATH)?;

  let response = blocking::get(content.trim())?;

  file.write_all(
    response
      .text()?
      .as_bytes()
  )?;

  Ok(())
}

fn main() {
  match run() {
    Ok(()) => {}
    Err(e) => eprintln!("{}", e),
  }
}

In order to apply our nice shorthand ? operator, we have the implement the From trait for each error type we wish to add to our custom enum. In addition, in order for main to be able to print out the error message, we must implement Display for our custom error type, matching on each variant.

This implementation solves the problem of dealing with multiple error types in a given function. Having our own custom error type also allows for other users of our application to deal with our various custom error type variants using a similar approach. However, as we shall see, we can avoid some of this tedium by bringing in a third party.

Enter snafu

The snafu library allows for the easy assignment of foreign errors into domain-specific errors while also adding contextual information. Whenever a foreign error type is encountered, we can easily pin it to our own custom typing by adding a source attribute.

For instance:

use std::{
  fs::{self, OpenOptions},
  io::{self, prelude::*},
};
use reqwest::{self, blocking};
use snafu::Snafu;

type Result<T, E = Error> = std::result::Result<T, E>;

#[derive(Debug, Snafu)]
enum Error {
  #[snafu(context(false), display("IO Error: {}", source))]
  Io { source: io::Error },

  #[snafu(context(false), display("Api Error: {}", source))]
  Api { source: reqwest::Error },
}

const PATH: &str = "file.txt";

fn run() -> Result<()> {
  let content = fs::read_to_string(PATH)?;

  let mut file = OpenOptions::new()
    .write(true)
    .append(true)
    .open(PATH)?;

  let response = blocking::get(content.trim())?;

  file.write_all(
    response
      .text()?
      .as_bytes()
  )?;

  Ok(())
}

fn main() {
  match run() {
    Ok(()) => {}
    Err(e) => eprintln!("{}", e),
  }
}

The Snafu macro implements all the necessary traits in order to begin using our custom error type in functions.

Crafting our type this way not only let’s us avoid implementing traits ourselves but also gives us the added benefit of easily adding in custom fields to our error types as contextual information.

This tiny example doesn’t show all of the functionality the snafu library has to offer, so I suggest you have a look at the official documentation.

Fin

Quality Rust code should never panic when it doesn’t absolutely need to, and knowing this you should make the best of the situation by crafting your own error types that other people can use in their applications. You can either make it tedious, by doing it by hand, or by using the nifty snafu library to do the job.

Higher-order-functions

Tue, 08 Jun 2021 04:00:00 +0000

The higher-order function is one of the many powerful and important concepts I have run into when dabbling in the world of functional programming.

Before diving into the details, it’s worth mentioning that in certain languages like Python or Javascript, functions are treated as first-class citizens. What does this mean? It means that functions can be stored in variables, passed around to functions or returned from functions, just as you would be able to do with a primitive or non-primitive data type.

Knowing this, a higher-order function is a function that either takes in a function as input or returns a function as output.

Examples of higher-order functions that you may deal with on a daily basis but may be unaware of the underlying concept: filter, map, reduce, sort.

Now that definitions are taken care of, let’s take a look at some examples:

Filter

Problem: given an arbitrary iterable, filter out elements whose value fail to meet a certain criteria.

The filter function takes in a single-argument predicate function (a function that returns a boolean value) and an iterable, and returns a new iterable that consists of each element in the passed in iterable that when passed into to the predicate function evaluate to true.

def filter(func, iterable):
  return [el for el in iterable if func(el)]

print(filter(lambda x: x & 1, [1, 2, 3, 4, 5])) # [1, 3 ,5]

Map

Problem: given an arbitrary iterable, apply some transformation to each element in that iterable.

The map function takes in a single-argument function and an iterable applies that function to each item in the iterable, returning the resulting iterable.

def map(func, iterable):
  return [func(x) for x in iterable]

print(map(lambda x: x // 2, [2, 4, 6, 8])) # [1, 2, 3, 4]

Reduce

Problem: given an arbitrary iterable, accumulate some transformation to each element in that iterable and return the resulting value.

The reduce function takes in a two-argument function and an iterable and applies that function to elements in the iterable, accumulating the result along the way.

def reduce(func, iterable, init=None):
  it  = iter(iterable)
  val = next(it) if init is None else init

  for el in it:
    val = func(val, el)

  return val

print(reduce(lambda x, y: x * y, [1, 2, 3])) # 6

As we can see above, we first deal with our initializer value, if none is present we use the first element in the list.

We then run through the list, applying the passed in func with our current value and each subsequent element in the list.

Composition of programs

Now that we’ve gone through some basic examples, let’s see how this pattern might be useful in a practical scenario.

Problem: write a program that outputs the sum of naturals, squares and cubes from numbers 1 to N where N is some arbitrary input

Seems easy enough? Without the use of higher order functions, we can simple write the program like so:

def natural(N):
  return sum([x for x in range(1, N + 1)])

def square(N):
  return sum([x ** 2 for x in range(1, N + 1)])

def cube(N):
  return sum([x ** 3 for x in range(1, N + 1)])

def main(N):
  for func in [natural, square, cube]:
    print(func(N))

if __name__ == '__main__':
  main(int(input()))

Notice that the three different functions are actually very similar, the only difference is the transformation of data we perform on each number in the sequence.

We can introduce a higher order function that applies this transformation to the data.

def summation(func, N):
  return sum(list(map(func, range(1, N + 1))))

def main(N):
  for func in [lambda x: x, lambda x: x ** 2, lambda x: x ** 3]:
    print(summation(func, N))

if __name__ == '__main__':
  main(int(input()))

Decorators

It’s worth covering another type of higher-order function, one that returns a function. In Python, the idea of a decorator is simply a function that takes in another function and returns a new one with some modified behaviour.

Here’s a simple example of a timer, a function that computes the time of a function call.

import time

def timer(func):
  def wrap(*args, **kwargs):
    start = time.perf_counter()
    func(*args, **kwargs)
    end   = time.perf_counter()
    print(f'Function {func.__name__} executed in {end - start:.4f}s')
  return wrap

@timer
def waste(n):
  for _ in range(n):
    pass

def main():
  waste(2000000)

if __name__ == '__main__':
  main() # Function waste executed in 0.0396s

As we can see, the function timer takes in a function as input and returns a wrapper function that modifies the behaviour of the passed in function by performing operations before and after the function call. Now the new function will embody this new behaviour whenever it is called.

Fin

Being able to treat functions as first-class citizens in your programming language is a powerful idea to wrap your head around and a worth while design choice when composing large scale and complex applications.

All of the functions written for this post can be found here -> https://gist.github.com/terror/4d86aaf49cc724d0bfe5af11d05da88e

Binary-search-for-bugs

Mon, 15 Feb 2021 05:00:00 +0000

I recently encountered a build failure in a dependency of a Rust crate I was trying to install which led me to be introduced to a git command called bisect.

From the official docs found over at git-scm.com, the small description under the name heading reads:

Use binary search to find the commit that introduced a bug

Sounds super cool right? Using this feature I was able to track down the commit that introduced the bug in less than 6 steps.

Here’s a general overview of how this command works:

Run git bisect start to start the bisect session
Specify a bad commit (usually HEAD) using git bisect bad <commit-id>
Specify a good commit using git bisect good <commit-id>

This will kick off the binary search and pick a new commit in which you can evaluate using git bisect good or git bisect bad, in between the two endpoints.

It is clear that using this method, over the linear scan approach, can save a tremendous amount of time in the quest for finding where a breaking change was introduced.

How-I-read-blogs

Sun, 15 Nov 2020 05:00:00 +0000

I have recently gotten into the habit of reading more blogs, in particular, ones featured on HackerNews.

Reading books, personally, has been a difficult habit to cultivate over time, which I mainly attribute to the periodic feelings of being “stuck” or bored on a certain set of books. Blogs, however, offer a condensed read making them efficient and often amusing sources of information.

My typical sourcing routine would usually just consist of scrolling through HackerNews, potentially seeing something that peaked my interest and reading it, or realizing I didn’t have the time and putting the url off in a .txt file somewhere.

Realizing the severe lack of efficiency present in this routine, I thought about how to make it better. First, by asking the question: what were the things I didn’t like?

Having to scroll through articles myself
Reading on mobile or desktop
Having no effortless like or archive feature

To address these problems I chose to try out Instapaper and build a simple tool called hackerpaper.

Instapaper has a feature that let’s you send articles to your Kindle, which makes reading blogs much more comfortable. Moreover, the built in like and archive features carryover to the Kindle as well, making it a seamless process of read -> archive or like -> repeat.

To solve the issue of having to scroll through articles myself and add them to Instapaper, as mentioned above, I built hackerpaper. This tool sends HackerNews articles to Instapaper with certain filters such as type, subdomains or a specific article id.

I run a cron job that sends the articles to Instapaper at 5:00AM and have Instapaper send those articles to my Kindle at 7:00AM.

The script is as simple as:

$ ~/path/to/python3 ~/path/to/hackerpaper -a <email> <password> -t topstories

To recap the routine:

Have hackerpaper send articles to Instapaper
Have Instapaper send those articles to Kindle
Read and archive those articles
Repeat

Useful-git-aliases

Wed, 02 Sep 2020 04:00:00 +0000

I really enjoy using aliases for git commands because a) I’m lazy and b) it makes me feel more productive.

Some basic ones I use

alias ga="git add"
alias gc="git commit"
alias gd="git diff"
alias gch="git checkout"
alias gst="git status"
alias gp="git push -u origin master"
alias gpo="git push origin"
alias gb="git branch"
alias gcb="git checkout -b"

These ones are set up in my .bashrc file so that I can just execute them as is, i.e gc -> git commit.

Some more complex ones (except for fetch) in my .gitconfig I enjoy:

f = fetch
fu = fetch upstream master
track = branch --set-upstream-to=upstream/master master
parent = "!git show-branch | grep '*' | grep -v \"$(git rev-parse --abbrev-ref HEAD)\" | head -n1 | sed 's/.*\\[\\(.*\\)\\].*/\\1/' | sed 's/[\\^~].*//' #"
lg = log --oneline
last = log -1 HEAD

fu and track are useful for doing things when I’m working on a forked repository

parent shows the parent branch of the current working branch

lg shows commit history one line at a time

last shows information for the last commit

Whether they make me more productive or not, I just enjoy using them and see no reason not to.

Chaining-comparison-operators

Tue, 25 Aug 2020 04:00:00 +0000

I recently came across this lesser known feature in Python and started to play around with it.

Being able to chain comparison operators is a rarity among most modern programming languages, which is a shame considering how elegant and intuitive it turns out to be.

Here are some examples:

a, b, c = 5, 10, 15
print(a < b < c)
# True

The code above is equivalent to:

a, b, c = 5, 10, 15
print(a < b and b < c) 
# True

And something a bit more complex:

a, b, c, d, e, f, g = 5, 10, 30, 20, 15, 0, 0
flag = a <= b > f < c > d is not f is g
print(flag)
# True

That is all equivalent to:

a, b, c, d, e, f, g = 5, 10, 30, 20, 15, 0, 0
flag = a <= b and b > f and f < c and c > d and d is not f and f is g
print(flag)
# True

You can see how tedious things might get. Python continues to amaze me with how beautiful and concise it can be.