Hugh Haworth - Developer Blog

Building this Blog

2020-08-20T00:00:00Z

Hey! This post is a little out of date! I had fun writing this PHP framework, but I've since moved to a static site archicture.

This blog is built using Edoras - a barebones MVC framework I built with laravel as my inspiration.

Here's a link to the source code.

I have been trying to learn laravel recently, looking at books, documentation and laracasts. These were all great resources. I'd especially recommend Code Smart by Dayle Rees. They showed me the benefits of laravel and how it creates something elegant out of the famously ugly language PHP.

I thought a fun way to learn laravel would be to try build my own implementation (with a LOT fewer features). It sounded like an interesting challenge. I want to be able to have a strong grounding before getting myself into trouble with laravel magic. I also didn't want to upload anything too big when building this site.

I started by taking a look at this great Medium article by John O. Paul which helped me build a simple router. Then I added the ability to make routes with parameters, view routes, and routes pointing to controller methods. I also created a tiny ORM as a way of interacting with the database using prepared PDO statements. Finally I built some simple authentication functionality.

Edoras doesn't have the fancy Eloquent ORM of laravel or the Artisan command-line tool, but it does give you a quick way to structure a PHP MVC application. It's small enough that it'd be easy for someone to hack into a shape they needed.

PHP Data Objects

2020-08-27T00:00:00Z

Back in the bad old days of PHP we needed to interact with our databases with raw SQL queries through the crusty old mysql extension.

We would build queries out of conctenated strings, attempt to escape user input, then execute these queries in our database. And then we'd pray our code wouldn't allow the user input to be assessed as SQL. What could possibly go wrong?

Things, of course, did go wrong. SQL injection. Data breaches all around. Companies going under from the fallout from usernames and passwords getting into the hands of attackers.

PHP data objects to the rescue!

PHP data objects (PDO) allow us to:

Write prepared statements which will automatically escape variables
Use the same API to interact with multiple kinds of database

Let's look at the code. You will need to know your database credentials (username password, hostname). It's preferable not to include these in your source code, and instead grab them from a .env environment file. It's out of the scope of this post to explain how to do this. Imagine we had grabbed our credentials and stored them in variables. Now let's establish a connection and store a reference to it in a variable:


$connection = new PDO("mysql:host=" . $myhostname . ";dbname=" . $mydbname, $mydbusername,$mydbpassword);

Nice. Now imagine that we're building a blog application and we have a "posts" table with three columns: an "id" primary key column, a"title" column, and a "content" column.

Let's create a prepared statement to get this data. This is will allow us to write a hardcoded query and specify parameters. These parameters will not be assessed as SQL:


$query = $connection->prepare("SELECT * FROM 'posts' WHERE id = :id;");

Now we haven't run the query yet, or passed it any VARIABLES. We've just prepared it. Let's give the statement some parameters, run the query, and grab some data!


$query->bindParam(':id', $id, PDO::PARAM_INT);
$query->execute();

You'll notice the third argument in the function I passed in PARAM_INT. This makes sure the parameter sent to SQL is assessed as an integer (as IDs usually are). Otherwise it would add quote marks and build the query as a string. You can also make a prepared statement without having to name the parameters, by writing question marks in parts you want to be dynamic. You just need to pass it an array with a count matching the number of parameters.


$query = $connection->prepare("SELECT * FROM 'posts' WHERE id = ?;");
$query->bindParam([$id], PDO::PARAM_INT);
$query->execute();
return $query->fetchAll();

What does this return to us? The default return is an array with both associative keys and integer keys:


echo JSON_ENCODE($query->fetchAll());
//{ "id":4,"title":"My title", "content":"My content","0":4,"1":"My title", "2":"My content"}

This is pretty weird. I'd probably usually prefer either an associative array or an std class object. Here's how we can get each of these instead:


return $query->fetchAll(PDO::FETCH_ASSOC);
return $query->fetchAll(PDO::FETCH_OBJ);

What if we want to write dynamically to different columns of the database by passing a key value pair to a function? Let's say we have a function with a $key argument. We can't use variables in the prepared statement, as these will be escaped into values. So we will have to go back to string concatenation:


$query = $connection->prepare("SELECT * FROM 'posts' WHERE {$key} = :value");
       $query->execute([':value' => $value]);
       return $query->fetchAll(PDO::FETCH_OBJ);

How do you keep this secure though? You could run a check to see whether the key matches a column name:


$query = "SELECT COLUMN_NAME FROM information_schema.COLUMNS WHERE TABLE_NAME = 'posts';";
       $keys = [];
       foreach ($connection->query($query, PDO::FETCH_ASSOC) as $result) {
           array_push($keys, $result["COLUMN_NAME"]);
       }
//This allows us to check for the existence of the key passed in
if (!in_array($key, $keys)) {
           throw new Exception("Keys do not exist in table");
       }

So there you have it. A very quick intro to the PDO prepared statement. Go forth and make queries!

Web Colour Deep Dive

2020-09-06T00:00:00Z

I've been trying to work on my design skills lately. As much as some might be able to respect good code behind the scenes, everyone can tell when a website or application looks good.

There are so many different aspects to good design: layout, typography, alignment, usability and responsiveness, but lately I've been taking a look into colour (yes, this is how we spell it in New Zealand).

Having a look at Adam Wathan and Steve Schroger's advice on the subject, we can see we will need more than just five nice looking hex codes from a colour palette generator to get the right colours for an application.

They advise having a whole lot of shades of grey to choose from, one primary hue, and a few sparsely used accent hues. And having multiple levels of saturation and lightness from these for different levels of emphasis.

When I'm picking hex codes or RGB colours developing an app I get slowed down by trying to work out different levels of lightness and saturation from a single colour. I haven't had a good workflow. So to get these different levels from a hex code or RGB, it's been between using tools like hexcolortool or trying to carefully move the right ranges in VScode's visual editor.

HSL Values #

One way to get out of this situation is to use HSL values. HSL stands for hue, saturation, lightness. Using these you can declare your hue as a number from 0 to 359, then note down a percentage for each saturation and lightness. For instance:


div{
   background-color: hsl(155, 60%, 60%);
}

Gives you a muted mint green colour. HSL values are supported in all major browsers and a superior way of defining colours compared to RGB.

You could use these values to set your primary and accent colour hues, then you can iterate on the levels of saturation and lightness throughout your app. You know that as long as the hues you have chosen work together, your application should look consistent in its colouring.

This is because the hues are so central in making colour consistent. It's also why RGB and hex codes aren't as good a format - because they conceal the hue a colour is based on.

Another solution:

Colour Grid App #

To quote Refactoring UI again "As tempting as it is, you can't rely purely on math to craft the perfect color palette".

Naturally, after reading this, I built a react app to mathematically craft a colour palette. Okay, it won't solve all your problems, and it won't give you a full colour palette, but it will start you off with some options.

The app creates 100 different levels of saturation and lightness based off one hue.

When I looked, I couldn't really find an online tool which worked like this. There are 16,581,375 colours to choose from on the web and I'd prefer to have some kind of system to limit these down but still enough options to flesh out a UI.

Using the colour grid, I have a defined set of options which I can use to emphasise or deemphasise different parts of an application or website.

Also, here are some things I learnt along the way:

How to find the Lightness of an RGB Colour #

The level of lightness of an RGB colour can be worked out by finding the average of the highest and lowest of the RGB values divided by 255 (the middle colour does not affect the lightness).

This will give you a decimal between zero and one which is the degree of lightness the RGB colour is.

Disclaimer: This technique does not account for luminance. Luminance is the inherent brightness of a hue. It's illustrated by the fact pure yellow looks a lot brighter to us than a pure purple.

My technique here will get you the level of lightness based on a programmatic measure of how close the RGB values get the colours to white or black, but perceived brightness is affected by more than just this.

Nonetheless, if you want to get an estimate of the lightness of an RGB colour, here is a javascript function you can use:


function getLightnessOfRGB(rgbString){
    //First convert to an array of integers by removing the whitespace, taking the 3rd char to the 2nd last then splitting by ','
   const rgbIntArray(rgbString.replace(/ /g, '').slice(4, -1).split(',').map(e => parseInt(e)));

    //get the highest and lowest out of red green and blue
   const highest = rgbIntArray.indexOf(Math.max(...rgbIntArray));
    const lowest = rgbIntArray.indexOf(Math.min(...rgbIntArray));

    //return the average divided by 255
    return (rgbIntArray[highest] + rgbIntArray[lowest]) / 2 / 255;
}

How to lighten an RGB colour keeping the hue the same #

To lighten an RGB value and keep the hue the same, you need to increase each RGB value by the same proportion of difference between the value and 255.

What does this gibberish mean?!

I think it's easier to explain with an example. Lets say we have this colour: rgb(0, 153, 255). That's a fully saturated cyan-y blue. Lets look at the difference between each RGB value and 255: red is zero so the difference is 255. Green is 153 so the difference is 102. Blue is 255, so the difference is zero.

Now when we lighten the colour, we need to increase each RGB value by the same fraction of our differences. Let's increase lightness by a tenth.

This means we need to divide each difference between the RGB value and 255 by 10, then add this amount to the RGB value. So to red we'll add 255 / 10 (25.5), to green we'll add 102 / 10 (10.2), and to blue we'll add 0 / 10 (zero).

This means the colour we end up with is: rgb(26, 163, 255). That's still the same hue, but a touch lighter.

Here's a javascript function which will do this for you:


function lightenRgb(rgb) {
   //Our rgb to int array function again
   const rgbIntArray = rgbString.replace(/ /g, '').slice(4, -1).split(',').map(e => parseInt(e));

   // Map over each array item and return the value, plus the difference between the value and 255 divided by 10
   const returnArray = rgbIntArray.map(rgbIntArray,value=>{return (value + ((255 - value) / 10))});

   //convert the array back into an rgb string
   return (`rgb(${returnArray.join()})`);
 }

How to darken an RGB colour keeping the hue the same #

Conversely, darkening is pretty similar. Just replace adding to the values to make 255, with subtracting to make zero:


function darkenRgb(rgb) {
   //Our rgb to int array function again
   const rgbIntArray = rgbString.replace(/ /g, '').slice(4, -1).split(',').map(e => parseInt(e));

   // Map over each array item and return the value, plus the difference between the value and 255 divided by 10
   const returnArray = rgbIntArray.map(rgbIntArray,value=>{return (value + ((0 - value) / 10))});

   //convert the array back into an rgb string
   return (`rgb(${returnArray.join()})`);
 }

So if you ever do need to lighten or darken an RGB colour keping the hue the same, those are some useful functions. Also give Colour grid or HSL values a try.

Thanks for reading.

Laters,
-Hugh

Upcoming project Plantr 2.0

2020-09-11T00:00:00Z

One of the first full stack applications I built was a gardening app. You could register, log in and the app would tell you how long your vegetables would have before they were ready to harvest.

My name for the app was originally "planter". But my partner said "lose the E. It's cleaner" in her best impression of Justin Timberlake/Sean Parker in the Social Network.

Of course Plantr has not exactly gained popularity of the website referred to in the title of the Social Network. That's okay. It served its purpose in teaching me a lot about building full stack applications. I'll go into some of the things I learnt:

What worked developing Plantr 1.0 #

I built Plantr with vanilla javascript and PHP. I used PHP's password hashing functionality to keep the passwords safe.

This password hashing is important, as generally people will use the same passwords for everything that needs a log in. So while it isn't too bad if someone hacks into a gardening app, the attacker might've also grabbed the password to your gmail and/or your bank's app. I used PHP's native password hashing function which uses bcrypt and it's own salting.

Essentially I was outsourcing this piece of security to someone a lot smarter than me. If you're just an application developer concentrating on developing a product, leave this kind of thing to the experts. Trying to roll your own security will undoubtedly end in tears. So I will continue to "stand on the shoulders of giants" (or software security experts) when building applications.

Plantr was a client side app with a simple PHP API on the back-end. I tried to keep the application simple by keeping most of the logic on the client side, and make my backend as dumb as possible. It would essentially just run CRUD (create, read, update delete) functions, and manage authentication. This emphasis on simplicity made reasoning about the application a lot easier and sped up my development time. Simplicity is something developers strive for and I'd really recommend Rich Hickey's talk on simplicity.

Also I had recently read "don't make me think" by Steve Kruger, which pushes developers/designers to optimise sites for usability. Using his philosophy I created a UI which was easy to use.

Watching out for password security, keeping a lot of logic on either back-end or front-end, and optimising for usability are all things that worked well building the application.

I would definitely use these techniques in future projects.

Now onto the uglier parts (please don't laugh too hard at some of these) :

What worked but was bad practice #

In Plantr I did a few things that didn't seem to completely ruin the functionality, but they are things that I would not do again.

Everything was "roll your own" #

The first thing was that I built everything in the app. The front end was completely vanilla javascript with zero npm libraries pulled in. The back-end pulled in one single library for emailing (PHPmailer).

This meant I wasted my time building a lot of functionality which definitely already exists out there in npm and composer. For instance, I spent a long time working out how to calculate the difference in days between two dates. Now I know of a great npm library called Day.JS. This would've been a lot quicker to pull in and hit the ground running.

This is why for Plantr 2.0 I'm going to build it as a server side rendered Laravel app (with a little client side javascript sprinkled in).

No database #

The next thing I did wrong was that I didn't use a database.

"WHAT? You built an app with authentication and didn't use a database?"

I know, it's bizarre. How and why did I do this? I wanted to concentrate on learning PHP and javascript and hooking up these two sides of the application. I didn't want to add a third variable/piece of technical debt to worry about. For this reason, I used CSV files - it was a little bit less of a jump for me to understand a flat file system than a database.

Next time I would definitely use a database for many reasons. Databases are designed for the use of web applications with security, concurrency and performance baked in. Also I wrote a lot of code to parse the CSV files into PHP objects. With the PDO statement API and a few SQL queries I could've grabbed this data easily by following conventions.

One other problem was that I had the filename holding user information baked into the source code. Thankfully, I never uploaded the source to Github, but if I had this would be BAD NEWS. An attacker might know the exact location of user data if they managed to hack the app.

With databases there is a tried and true convention of using a .env file ignored by git to store your database credentials which is a lot better.

What didn't work #

All Static classes #

That's right. I built the entire back-end application with 100% static classes.

As I said before, I was trying to keep the application as simple as possible. I was also getting interested into functional programming and the habit of getting rid of state and side effects. So I tried to use all static classes to get towards this ideal.

It didn't work.

If I really wanted to double down on functional programming, I should've been using good run-of-the-mill functions. These would be far easier and quicker to compose than static methods on a class. Using static classes I was also missing out on one of the most useful concepts in object oriented programming. That's right, I was missing out on DEPENDENCY INJECTION.

In my more recent programming experience I've "heard the good news" of dependency injection. The idea is you pass the class what it needs (usually in a constructor), you tell the class what it needs to do (in functions) then you grab from it what you need. And we don't have to worry at all about the implementation details throughout this process. It's all hidden away in the object itself.

An example of where I could've used this is in Plantr was when I was processing CSV files I had two different classes for the users table and the plants table. They both had a lot of the exact some CSV CRUD functionality copied and pasted in. If I had coded this properly I would've had a "CSVModel" class. This could have all the CRUD actions based in, and the application could simply pass the filename of the CSV file to the object constructor. That would mean being able reuse a whole lot of code.

The other benefit of dependency injection is it makes testing a lot easier. The tests can pass the object the data it needs, and test whether this data will produce the expected output.

This brings us to the next thing that didn't work:

No tests #

I did actually originally try to write Plantr with test driven development, but I gave up very quickly. And it cost me.

Testing takes some setup, but it will pay for itself in time. Parts of your application can break at unexpected times. Things go wrong. We are only humans and will definitely make mistakes. Half the time as developers, we will be console.logging, writing logs to files and manually testing to make sure our application is working anyway. Why not instead turn this into a test you can continually run from one command?

When building Plantr I would constantly break things. If I had written some tests in the program I would've know exactly where and when the breakages were happening.

Having no tests did not work.

Plantr 2.0 #

So I'm going to take what I learnt from this experience and re-build Plantr. This time as a laravel application hosted on heroku and here's what I'll do:

Use tried-and-trusted security techniques
Aim for simplicity in the application logic where possible - keep most of the logic either on the server or the client
Aim for easy usability
Use libraries and frameworks if the cost isn't too high
Use a database (of course)
Leverage object oriented programming/dependency injection (no static classes)
Test!

Let's see what I can learn this time. I'll post a link when it's complete!

Eloquent Models - First Impressions

2020-10-05T00:00:00Z

I've just started building my gardening application and thought I'd note down a few of my first impressions getting into using Eloquent models. Here are a few of my thoughts:

They're easy to work with #

Yup, I managed to do what I would had done with vanilla PHP and home-rolled CSV functions in 5 hours in about half an hour.

Most of the code is already written #

When you're using eloquent models you barely have to write any code at all. The only thing the class really needs to do is extend the Eloquent Model class and you get a whole bunch of functionality. Sometimes you'll also need to define some relationships between different models - eg. an author owning a blog post.

The main hassle with eloquent models is writing the migrations which will generate the database tables and rows you need. It was quite slow-going looking throught the laravel documentation to find out what to write to create data of a certain type. I don't think this is laravel's fault or anything - there is always going to be some complexity in defining the data that your application needs.

Protection from mass assignment #

Laravel does not let you set a big load of database rows from a request without your careful supervision. You can't use an array to create or update an eloquent model unless all the properties you are setting are set to "fillable". Alternatively, you can set some properties as "guarded". This will mean the default is that you can fill create/update the model with an array. But the properties set to "guarded" can not be filled with an array.

In my applications I think I will be using the "fillable" model where I expressly declare anything that can be filled with an array. I also think I might avoid using arrays to fill up models anyway so I can keep everything explicit about what is going on. We'll see where this takes me.

Batteries-included React

2020-10-16T00:00:00Z

I went to my second ever Javascript meetup last night. It was basically a free lesson from a senior frontend developer based in Wellington. I felt like I should be paying more for all the advice I was getting.

... Hopefully the guy who hosted the meetup doesn't read this and start charging!

We were taking a look into building a React application. So far I have mainly been learning React in isolation (in fact for a while I was avoiding other tools to focus on the fundamentals). But in the meetup I got to see some of the standard libraries and tools used around React.

Here are a list of the React-adgacent tools, libraries and techniques we looked at:

These all have varying uses and levels of popularity. I had used Prettier and ES lint before. And I generally use npm instead of yarn. But it's good to know there's another option out there.

It took quite a while to set everything up, but we were discussing all of these elements of the ecosystem as well so I got to hear about each item's benefits.

I'm definitely keen to start pulling from this list when making future projects.

Next.JS #

Another option that is starting to sound increasingly tempting is Next.JS. I think I was worried about it earlier on because it sounded like quite a complex system but it doesn't sound like the developer experience is bad at all. And it sounds like a really good way to get performance out of React.

Next.JS can be used as a full static site generator. You program it to fetch data from a source at build time and add the data to your components with a function called "getStaticProps". This means you get all the performance and security of a static site but you get the composability of React components.

It can also be used to server side render pages if you need to grab dynamic data at each request from the user. And it uses a similar mechanism to turbolinks to grab pages from the server without a full page refresh.

Next.JS is an opinionated "convention over configuration" framework. Some are saying it might be "the next Ruby on Rails". I definitely want to give it a try.

React on a Budget with Lit-html and Haunted

2020-11-17T00:00:00Z

I love using React.

I love not having to query the dom every time I have to show or hide a modal. I love being able to know where the source of truth lives in an application. And I love hooks! A bunch of useful closures with escape hatches to the imperative world? Sounds good!

React is based on a state-driven UI concept. This concept is powerful enough that React has been inching ahead in the front-end framework wars - even after the drama about JSX in 2013.

But I'll tell you what I don't love:

I don't love 2 megabyte "hello world" apps. I don't love having to buy into a build step to get code to run on a browser. I don't like having to download hundreds of webpack dependencies to get a little interaction happening on a website. I guess this isn't really the use case for React. But some sure treat it like it is!

What if we could get some of the goodness without the bad?

This is what lit-html gives us. It's a new UI Library from the folks over at google's Polymer Project.

It's a LOT smaller than React, and can be used without a build step. How does it manage this? It uses native web components. These are basically a way to wrap HTML templates into a custom angle brackets tab NATIVELY - using functionality which is built into the browser. It also leverages JavaScript template literals (those are the strings with the back-ticks).

You can find lit-html compared to a number of other components on the Web Components Dev Website.

Now it's kind of could be a barrier how different the lit-html API is from React. But if you take a look at the list of web components libraries in that link, you might find one called 'haunted'.

Hooks in Lit-HTML with Haunted #

An awesome way to leverage lit-HTML is with Haunted. This is a wrapper library around lit-HTML which uses the React hooks API. This means that you can utilise all of your favourite React Hooks patterns, but without having to pull in the entire size of React.

I haven't had a lot of chances to experiment with Haunted but here's a cheeky little Codepen to check out how to get from zero to interactivity with Haunted.

I'm excited to continue using Haunted in future projects.

Tailwind for interactions - Alpine.JS

2020-12-07T00:00:00Z

As you can probably see from my earlier post about Lit-HTML and Haunted, I've been looking at alternatives to React for adding interactivity to pages without a whole lot of technical debt and without shipping a gigantic package size to the client.

One of the great ways of getting this done is with Alpine.JS. A framework by Caleb Porzio - a developer from the Laravel community. Alpine JS is pretty different from some of the other Javascript libraries/frameworks out there in a few ways I'll get into:

Firstly, Apline is designed from the ground up to work in a traditional server side rendered (SSR) app - eg an application using frameworks like Rails, Laravel or .net. You can have a web app just pushing HTML to a browser and you don't need to scorch it all to the ground before adopting Alpine. All you need to do is pull it in from a CDN and get started - just like the old Jquery days. No build step.

If you've used React before, you'd be used to pulling HTML into your Javascript and using babel to convert it back into React components. This is handy, as the logic is colocated with the markup and you can see exactly what part of the markup is being interracted with.

Alpine goes the opposite way. Instead of pulling HTML into your JavaScript, you can instead manage all your interactivity inside your HTML - and you don't even need to worry about creating a new JS file. This is especially good in server side rendered applications - which are literally designed for producing HTML based on data.

In my experiments with Alpine, I've also found that it's pretty close to being able to do an almost redux - style state management. You can have your child elements create events which are listened to by the parent, which can in turn run these events and their payload through a reducer function to update the state - which will then filter this data down to the child elements.

Also, the child elements of your Alpine component can react to the component state in a number of handy ways. You can bind their text or innerHTML to values held in the state. You can decide whether they get shown or not based on a vlues truthiness. You can render out a number of elements based on an array/list by using the template tag. And you can use "x-spread" to reuse logic.

Alpine also has built in transitions for when elements get added and removed from the page which is pretty tasty.

So far in my gardening app, I've been using Alpine.JS for a few different things: form validation (with Iodine helping out), modals (for deleting and changing some items), and some transitions.

I'd definitely recommend giving it a try.

2020 to 2021

2020-12-31T00:00:00Z

2020 was the worst year ever for a whole lot of people. I got off super easy being from New Zealand and having steady employment which I could do from home during lockdown. And thankfully we only had the one lockdown, unlike a whole lot of other countries. So I feel very lucky to have done anything at all in 2020.

Some things I learnt in 2020 #

I've learnt a WHOLE lot in 2020. Throughout 2019 I was mainly learning the fundamentals of the web: plain old HTML, CSS and JavaScript, and a bit of PHP and Jquery too. 2020 for me was about learning frameworks, tooling, hosting and the backend.

In March 2020 I pushed to this site. For the first iteration of this site I used static HTML files and a few simple links. This helped me learn about FTP, servers, and DNS.

Then I started reading about React and did a React course. I installed node and started taking a look at webpack, npm and babel - all these heavy powerful JavaScript tools. Like moving from a chisel to a jackhammer. I learnt about the power of React hooks, composability, and functional programming.

I started making commits and pushes using git and github.

I learnt a whole lot about CSS too. Around the middle of the year I did all of Flexbox zombies. I learnt a little bit of colour theory and how to design with colour. In the process of learning this I built a React app to programmatically generate a monochromatic colour scheme. I even got a post up on CSS tricks about the JavaScript functions I used to calculate the changes in hue and saturation. Design is something I read a lot about and something I am definitely keen to continue working on.

Then I started learning about Laravel - the style of building web applications pushing server-rendered HTML. I'm still working on Plantr 2.0 today, which is going to be a laravel app. I didn't want to use JQuery so I pulled in Alpine.js. This felt similar to React, but was very unobtrusive in the server-rendered world.

I thought the best way to learn Laravel would be to rebuild it - albeit with a lot fewer features. So I made Edoras - a little mini PHP framework for blogs based on the Laravel syntax. In the process of this I started reading into SQL queries and databases.

In August I changed up my portfolio site. I rebuilt it to use Edoras and started this blog. I tried to use some of the fundamentals of design I learnt this year. Finally I started getting into Codepen and making tiny little demos on there to get my feet wet with a whole lot of different technologies.

Looking into 2021 #

My new years resolution is to read 12 books in 2021. One per month. I'm going to make them half technical books and half non-technical books. Who knows whether I'll be able to do this when society is going to fall apart and we'll be living in a Mad Max style post-apocalypse if 2021 is anything like 2020. I'll write a post about each book and what I've been learning in the technical ones.

As for what tech to learn: I think after I finish my gardening app I want to start digging further into the React ecosystem. Server rendered components sound pretty interesting so I'd like to try these out when they get properly released for production.

I'd also like to learn Next.js. It looks like the tutorial on the Vercel website is really good so I think I'll start with that and see where we go from there. Next.JS sounds like it will combine my knowledge of React but with the battery-included-framework power of Laravel.

In the React ecosystem I'd also like to get into Styled Components, GraphQL (apollo and urql), React Router, React Transitions, material-ui, jest, xstate and immutable.js. That's a lot. And I'm definitely not going to be able to dig super deep into all of these, but I'll find out what interests me in the process. Also while I'm at it let's throw Node in there too (or maybe deno if it looks like hosting options are coming out).

I also want to get started with TypeScript, as it's sounding like a good way to avoid bugs and get some nice autocomplete going in VScode. Finally, as I love trying to make things look pretty, there'll be more CSS. I'm interested to try out Tailwind, and try to keep up with all the new developments we're getting in browsers.

That's probably enough to keep me busy.

Dropping Plantr 2.0

2021-01-12T00:00:00Z

That's right you heard me!

Plantr 2.0 Released! #

Maybe it's more of a 1.0 and the last one was a 0.0. 🤔 Well anyway, this is my latest and greatest project, Plantr.

I had heaps of fun building Plantr. Here's the stack it's built on:

Heroku for hosting (the free version at the moment so a slow initial pageload)
AWS postgres database (configured through Heroku)
PHP
Laravel
Alpine.JS for front-end interactions
Vanilla CSS for styling and animations
Inkscape for illustrations
Blade Icons to integrate the SVGs into Laravel
SVGO for vector image optimisations
Squoosh for raster image optimisations and conversions

What I've been learning building Plantr #

I put a whole lot of work into making it an awesome user experience using animations, illustrations and icons. I used Alpine.JS a lot on the front end, because it integrates so easily with laravel.

Here's a link to the source code, and a tweet with a preview.

I was mainly trying to learn about Laravel building this app. I dug into eloquent models, blade templating, form validation and authentication. I actually editted the composer.json to pull in a library which is something I've been too scared to do so far. I also got into illustration. I'm really proud of how the icons I drew on inkscape turned out.

Looking forward #

There always feels like more to do with side projects. Some things on the todo list include getting some more functionality happening with regards to different climates and different tyoes of planting.

Also it would probably be good to add a few more options to the plants. Also I said I would write some tests for this project and I haven't written them . . . yet.

Finally I want to dig more into asset compilation. The CSS and JS in this project isn't even minified. Although I did configure webpack from scratch for a React project I wasn't super sure how to do it in a laravel friendly way. Mixing the Node and PHP environments just seemed a little bit weird to me - even though it's the norm. I enjoy doing things in a simple way so I thought about pulling in grunt or gulp to do some quick CSS minification. But there just aren't enough hours in the day. And always more stuff to learn!

I think I'm going to leave Plantr alone for now and look toward a new project. We'll see what happens

Easy Node Live Server

2021-01-17T00:00:00Z

There are a thousand different ways to get a server up and running in different environments.

For my Laravel programming I was using laragon, which has been working a charm - super easy to set up and configure.

I've created a few create-react-app projects. This is awesome too, however it takes about 10 minutes for me to download all the packages for webpack, the live hot module reloading server, and babel. In the end it's extremely powerful, instantly checking and reloading the code as you type it without changing the client side state.

This weekend I was working on my CV. I built it as a single webpage taken from the front page of my portfolio and converted to a PDF using html-pdf-node.

When I was working on it I really didn't want to have to pull in the entirety of webpack for a dev server. Here's a look at some of the node tools I tried:

HTTP-Server #

First I used HTTP-server. This was really quick to download and provided a very easy server. Really good for if you just want to see what some files look like on a browser without getting errors from trying to pull JavaScript in from a local filesystem.

The problem with this was that I had to continually press Cntrl R all the time to view my changes. I had to do this using laragon too, and a number of times I've been developing. I know there are better ways to do this and I wanted to instantly see my changes with live reload, so I tried:

Nodemon #

Nodemon's a tool that restarts your server whenever files are changed. Unfortunately I couldn't work out how to hook it up with http-server. Maybe it was because I was using it locally instead of globally and the examples were showing how to use the global install. Maybe I was just pointing it at the wrong file or not providing a proper server entry point.

Either way, I ended up giving up. Although I might try this one again some day.

Live-Server #

For me, this was the one. I just installed it globally after what happened with nodemon. I just ran it in the command line, and it just worked. I editted an HTML file and pushed save and watched it reload. I editted a CSS file and watched it reload.

Perfect.

Going static

2021-01-25T00:00:00Z

Over the last week or so I've migrated this website and blog from a custom PHP CMS I built to an Eleventy build hosted on an Azure storage container.

So why did I go static?

I did get partly inspired by this Eric Meyer post about going static. He's mainly addressing government agencies in a call for them to improve their web services to provide accessible information in the time of Covid 19 - So not a whole lot to do with my little developer blog! But it underlines some of the benefits of static sites. Here are the main reasons I went static:

Why go static? Performance #

Moving from old-school PHP shared hosting to static means you get the benefits of the CDN (content delivery network).

A CDN makes copies of your site and distributes them to servers around the globe. It means that people visiting your site pull the information from an closer source regardless of where they are in the world, whereas my old hosting company were based in Sydney. CDNs also use caching to increase performance.

WordPress sites can have CDN level caching if you know what you're doing and pay more for the set up, but static sites have CDN hosting almost as the default.

The other reason the performance is better is that you don't have to boot up a PHP instance and grab data from a database on every request. PHP is reasonably fast, but these steps definitely take time.

Why go static? Money #

This is one that doesn't get talked about as much as I'd expect. Maybe because everyone on the tech industry is so damn rich, but static hosting can be cheaper than PHP hosting. So far the hosting has cost me nothing with Azure. But after a year they start charging by the amount of data they send over the wire.

The thing is, this site would have to be visited hundreds of thousands of times before the costs would match what I've been paying to run a PHP server and MySQL on shared hosting. Another benefit is the free SSL I get from Azure which would've cost about $100 a year from traditional hosting.

Why go static? Deployment and developer experience #

With the previous PHP host, I needed to use FTP to upload my site. Most of the time this was okay but there were times that I forgot to upload a file and it caused the entire site to crash. I also made a simple misspelling in my database credentials once and I think it took me two hours to find and fix it.

When setting up a dev server for my PHP site, it would take a while to set up which folder to point my dev environment, and I still didn't get any live reload.

Now I can develop with Eleventy using the built-in dev server. Then when I've finished, commit those changes and push to github. Then (and this is the cool part) a github action will automatically push the site up to Azure using the Azure cli. I used this guide from the Azure docs to get it done. Although for some reason it does send me a warning every time - even though it seems to work.

This is still a far better experience than the PHP deployment experience i had.

I would definitely recommend going static for a simple blog site and I'm excited to keep using Eleventy as a way to build performance websites.

HTTP Caching is a Superpower

2021-05-11T00:00:00Z

What do people mean when they say “use the platform” or “web fundamentals”?

The web is well designed: It’s reliable, it’s consistent, and it’s open. Some sites built on it aren’t so great. But we keep coming back anyway. I’d maintain that the original design of the web is, and always has been, pretty damn good. If we have a deeper look into its basic building blocks, we can improve our websites.

The web got started with a few fundamental features: Hypertext Markup Language, the Uniform Resource Locator, and what we’ll be looking at here: the Hypertext Transfer Protocol. These features give web browsers the ability to fetch and present data. But they can also instruct the browser on how to perform these functions. Why should you care about these specs from the ‘90s? Here's why:

We can continue to improve experiences using features the web has provided for decades.

Let’s think about how we become web developers. Usually it’s something like this:

We start as consumers of the web. We learn how to use browsers, including the “back” button and the hyperlink. We learn what URLs are by typing them into the address bar.
One day we decide we want to learn how to make websites. The roadmap usually tells you to start with authoring with HTML, then styling with CSS, then building interactivity with JavaScript, (and/or producing dynamic documents with server-side languages).
We end up pretty far ahead before looking at a very basic part of the stack in any meaningful way: the humble hypertext transfer protocol.

HTTP 1.1 #

I want to take us back to 1997. Candle in the Wind is ringing in your radio. You’re greiving for the Peoples’ Princess. You’re wearing low-rise baggy jeans. Bill Clinton is starting his second term. Apple is a computer company that seems to have lost the war against Microsoft. A mobile phone is a suitcase with a phone attached to it.

This was the year the RFC for HTTP 1.1 was officially released. Why was it released? It tells you in the purpose section:

“The first version of HTTP, referred to as HTTP/0.9, was a simple protocol for raw data transfer across the Internet. HTTP/1.0, as defined by RFC 1945 [6], improved the protocol by allowing messages to be in the format of MIME-like messages, containing metainformation about the data transferred and modifiers on the request/response semantics. However, HTTP/1.0 does not sufficiently take into consideration the effects of hierarchical proxies, caching, the need for persistent connections, and virtual hosts.”

While Britain was handing keys of Hong Kong government offices to China, a bunch of developers were working hard to improve HTTP: The very protocol Facebook uses today to send you misinformation and bizarre craft videos.

There was a whole lot of good stuff that came from the HTTP 1.1 proposal. We got persistent TCP connections. This meant less work handshaking between computers. We got PUT , DELETE , TRACE , and OPTIONS for request methods. We got the Content-Encoding header so we could compress the data over the wire. We got the ability to have one IP address serve multiple domains.

And on page 100 of the proposal you can find we got a new header called “cache-control”.

The Cache Control Header #

This HTTP header will make your site faster. It'll improve user experience. It’ll give you better SEO. It's been doing all this since before 9/11 and people don't talk about it enough. You don’t need any specific framework or language to use it. It’s not “hot”. It is an underrated building block of the web platform which is probably offsetting the carbon footprint of Ireland (an estimate I’ve made based on nothing).

How can we activate this superpower?

First, you’ll some kind of access to the server-side of your site and a way to specify the HTTP headers of what your application sends. If you’re running a PHP/Ruby/ASP.NET/Django site this shouldn’t be too hard. For each of these you’ll need to find out how to specify content headers in your HTTP response.

Here’s some links for how with each language/framework:

HTTP header function in PHP or the header method of the response class if you’re writing Laravel
The action dispatch class in Rails (specifically the conditional get module)
The Headers property of the http response method class in ASP.NET or with this guide
The cach control view decorator in Django

If you’re going for a JAMstack/static site approach it might be a little more difficult.

I could attempt to find out how to set an HTTP header in AWS but that might take a lifetime. I expect developer-friendly companies like Netlify, Vercel, and Cloudflare provide easy ways to set these headers. I’m a masochist, so I host my static site on an Azure CDN and I found the way to set headers is using their Rules Engine.

Also, if you’re a quote unquote “real” programmer - with a grey ponytail, history of taking LSD in the ‘70s and a loathing for writing in any language other than C - you can configure HTTP headers with Apache, or with Nginx.

What does HTTP caching do? #

HTTP caching allows us to stop browsers re-fetching information they already have.

This is why server-rendered websites will usually load faster on the second page than the first. When I was learning Laravel my sites always took about two seconds: On the first page, on the second page, the third. Always. I was always re-fetching CSS, images, other assets. Every time I clicked on a new page my browser would ask the server for everything all over again. This includes fonts, where I can’t imagine any case where you’d want the same URL to ever return a different font.

How did other Laravel developers get their pages to load so fast on the second run? How did they prevent the page re-fetching everything? Did they always use turbolinks to fetch each page using JavaScript and rerender? Were they always pulling in Vue and fetching JSON data?

The (far simpler) answer was HTTP caching. You can see it in action if you head into your browser’s developer tools, jump into the network tab and refresh a page. Under “Size” you’ll usually see a bunch of files which say “disc cache” or “memory cache”. These files are the beneficiaries of somebody, somewhere setting a Cache-Control header.

You can see the amount of data they’ve saved going across the wire by keeping your developer tools open and doing a hard refresh (Control + F5 on Windows, Command + F5 on Mac). You might have even used a hard refresh before, but not known that this is what a hard refresh does: It’s a refresh which ignores the browser cache.

You don’t have to tell the browser to do this using JavaScript. This logic is baked into the platform.

What should we cache? #

We first need to decide what to cache.

We don’t want any dynamic content to be cached, otherwise when we refresh the page or click on a link, we might get stale data.

For example, let's say we're on a neopets forum. We write a comment. Something like: “The most beautiful neopets item is the fire and ice blade!”. In a traditional server-rendered forum we’d get redirected back to the original post. And we’d want the HTML page to be re-fetched from the server. Otherwise our contribution to the conversation won’t be visible to us - the browser would have shown us the cached file instead of re-fetching the updated page. There is a way we can cache the file and check with the server whether things have changed with something called an Etag - more on this later.

However, there will be many things on your neopets forum post which don’t need to change when your browser makes a new request. The styles won’t need to change and can be cached. This is one of the original arguments for separating CSS files from HTML. The fonts will stay the same. The images will usually stay the same. You might have some JSON data that you can trust you’ll never need to change - for example I built a phonetic alphabet quiz app and I can trust that the phonetic alphabet JSON won’t change. If any of these do need to be updated, you also do have the option of serving them from a different URL.

A more controversial issue might be your scripts. If you find a terrible security flaw in your JavaScript bundle, you won’t want your user’s browser to touch it with a ten-foot pole. If dangerous scripts get cached, your user will be stuck on the other side of the world with an evil gremlin in their browser. For this reason, if you are caching JavaScript, it’s a good idea to version your bundles to create new URLs for updates. You can set up build tools to do this, and there will be a thousand Medium articles to show you how.

Essentially, we want to cache an asset if we can comfortably say we’re not going to need to update it. and if we do need to update it, we can point to a different URL without affecting the user experience.

How do we use HTTP caching? #

The number one way to use HTTP caching is to set the Cache-Control header to a max-age value. Here you can define (in seconds) how long you want the browser to keep using a cached version of the resource.

If head to the Settlers of Catan website, we know we’re about to see some medieval fonts. A quick dive into the source and we can see they’re using Minion Pro. Nice. Times New Roman of the sophisticated. On the network tab, after reloading the **checks notes ** 9MB of assets?! We can see Minion Pro is being fetched from the following URL:

https://use.typekit.net/af/ea8d85/0000000000000000000151d1/27/l?subset_id=2&fvd=n7&v=3

If we click into the headers of the response, we see:

cache-control:` ` ****``public, max-age=31536000

60 seconds x 60 minutes x 24 hours times x 365 days = 31536000, so the font host is telling our browser to cache the font for a full year.

We know what the max-age means. What about the public? Well, here’s where we find out that your browser isn’t the only thing that caches the request. When we make HTTP requests, they will usually pass through proxy servers - servers inbetween the client and the server.

If we set cache-control to public, these proxy servers will also cache our requests. Even if the browser doesn’t find the file in its cache, the proxy server may prevent a request from getting all the way to our server. This saves our server compute-time and effort. We may not want proxies to store private information, so we have the option to set cache-control to private instead, which will only allow the browser to cache the response.

What other instructions can we give using the cache-control header?

If we don’t want the resource to be cached, we have the option of using the cache-control:no-store option. This will make sure that CDNs, proxies and browsers don’t store something we don’t want them to. Good for anything that we know will change from request to request.

This is not to be confused with cache-control:no-cache which doesn’t actually stop the browser from caching. Instead, this tells the browser to check every time with the server whether the resource has changed, and only if it hasn’t changed will it use the cached version. This begs the question: How does the server tell the browser whether the resource has changed?

This brings us to our next topic:

Etags #

Etags are still another way to stop a resource from needing to be downloaded.

An Etag is an arbitrary string which the server can choose to provide with a resource. It will usually be a hash generated from the body of a response (with something like MD5) or instead generated from the last-modified-time of a file. It’s up to the server how this is done.

The browser will store the Etag when it first downloads a resource. Later, if we refresh our website and the following requirements are met:

browser has the file in its cache and either
the max-age has expired or
the cache-control from a previous request was set to no-cache

Then the browser will send a request to ask for the resource again with the previously downloaded Etag and an If-None-Match header. The server will then generate a response with an Etag. If this new Etag matches what the browser sent, the server won’t send the whole response back. Instead it will sent a 304 response with no body. This is HTTP-speak for saying “The file in your cache is fine” and the entire response will only be about 80 to 100 bytes. A little bit smaller than when Jason Miller writes a DOM library.

This Etag method of caching isn’t as performant as when the browser pulls from its cache without a request - we still have to send an HTTP request over the wire and ask the server to check if anything has changed. The good part about it is that the server has fine-grained control over when to bust the cache when things have changed. So using etag responses, we could actually cache more dynamic content, like HTML files, as long as we’re finding a way to generate a new Etag based on our response body or last-modified-time of a file.

Conclusion #

You now have a simple, platform-reliant way of preventing unnecessary requests. You have another tool in your belt to save your users time and money. Also you’ve got a way to save a little carbon from being released into our atmosphere to power a server farm. And you can use this tool with any style of website: static file sites, single page applications, and server rendered applications. It’s a superpower.

But don’t take it from me, here are some smarter people than me who’ve written on this topic:

And if this isn’t enough power for you and you want even more out of your browser cache, you can take it to the next level with Service workers.

Reviving an old laptop with linux

2021-07-21T00:00:00Z

In 2016, back when we could freely travel without being a virus vector, I went on a 6 month trip through Europe. I needed a cheap crappy laptop to check emails, read books, and go on the internet.

I ended up with the Acer N15P2. A little plastic thing with a tiny screen and a single core Intel Atom (slow) processor. It was nice and light and although it wasn't the fastest computer, it was exactly what I needed.

It's been 5 years now and the old dog hasn't been used a lot. It's well and truly slowed down. There was nothing on there that I hadn't already copied somewhere else, so I decided it's time to see whether it would run faster on the world's favourite open-source operating system. I had know idea what weird world I was getting into.

Deciding on a distro #

First I needed to decide on a Linux distro. I wanted something that could run on a low powered computer and found the cutely named puppy Linux which was designed to run on slow computers. Step 1 was downloading the distro from their website. But it turns out there are a lot of different types of puppy Linux to choose from. After doing little-to-no research i went with a version called fossapup.

Using Rufus to set up the USB drive #

The next thing to do was add the distro's ISO to a USB stick. You can't just drag the ISO file onto the disk though. You need to use a tool to convert the USB stick into a bootable drive. A good windows tool to use here is Rufus. The USB stick needed to be completely wiped to be converted into a bootable disk so I pulled off the ebooks I had stored on there.

Now there's two different ways to set up the disk: GPT and MBR.

It looks like GPT is a newer, better way of storing the data, however older computers use MBR. My computer being 5 years old was still new enough to use the GPT version though so if youre following along (godspeed if you are) you probably dont need to worry🤷.

Then it was time to plug the USB stick into the laptop and boot from it.

Booting from a USB drive #

Here is where we start interacting with firmware 😬. At this point you need to tell the computer that you don't want to boot from the normal disk with the previous operating system, and instead boot from the USB stick.

To get to the setup for the firmware on windows you can use instructions from this guide.

From the firmware settings you'll need to get to advanced settings and turn secure boot mode off. I'd love to provide step by step instructions on how to get there but there seems to be differences in the BIOS/UEFI menus between different computers. You'll also need to enable the F12 startup menu on some computers- my laptop didn't have an option to do this but it automatically worked.

Now that l’d set up the firmware settings to allow me to potentially ruin the laptop, I restarted it and started bashing F12 until a menu opened with two options: windows boot manager and my USB. I selected the USB and I was in fossapup!

Fossapup #

Fossapup seemed really cool - it was super small, super lightweight, and worked absolutely fine with my touchscreen and keyboard. It had a bunch of open source programs for pretty standard uses. I love how simple all of the names of the apps are too - “write”, ”edit”, “listen”. Listing the exact thing we want to be able to do with each app.

Unfortunately, you can't actually run puppy Linux without some external drive to boot up. So it wasn't going to be quite the windows replacement I wanted. I didn't want to have to keep a USB stick to start the laptop. So it was time to try another distro.

Other distros #

I tried to install a few different distros: linuxlite, absolute Linux and then Ubuntu, but none of them would show up as an option on the old acer’s boot menu.

I had already deleted windows when fossapup was running (yolo) so what the computer was booting into was something called an EFI shell. I tried typing a few commands but I had no idea what I was doing so got nowhere.

Bootia32.EFI #

I was ready to throw the laptop into the nearest stream when I found some post deep on a forum showing some commands in a shell dialect which looked absolutely nothing like bash or powershell. Apparently UEFI uses a DOS style command line language. So I tried a few commands and found the USB stick and the “BOOT” folder within it.

I tried to run the boot command and it gave me an error message that you can't run a 64bit boot method on a 32bit UEFI system. This was quite weird as the laptop has a 64bit processor? I even downloaded 32bit Ubuntu and tried to boot off this - but no dice.

Another deep dive on the internet and it turned out a lot of cheap laptops from the mid 2010s used 32bit UEFI systems with 64 bit programs.

So how do we get around this? well yet another search through the internet yielded some results. I found this life-saving Medium article. It turned out the ISO we’re booting off needed something called a bootia32 file which you can download from this github repository from a guy called Hirotaka Niisato. I’m not sure what black magic he used to create this file but he is a hero.

To use the bootia32 file you need to place it in the EFI > BOOT folder. So this means you need to first extract the ISO using a program like 7-Zip, add the bootia file, then rebuild the ISO file using a program like ImgBurn.

Reincarnation with Ubuntu #

After doing this I could boot into Ubuntu! Aftter booting from the USB it allowed me to install Ubuntu onto the laptop’s main disk and it’s been running well since. Ubuntu comes with a host of programs including the Libreoffice suite, Mozilla Firefox, and a music app called Rhythmbox. Although Ubuntu isn't the most lightweight version of Linux, it does seem to run smoother than the heavyweight windows 10. And now I can sit down on the couch and read ebooks from it, which it was too frustrating to do before.

It also means I have a computer to natively practice bash scripts and writing with C, as linux ships with GCC and grep. And generally it feels kind of good to be using a free open platform to get things done.

I haven’t had any problems with the laptop’s hardware other than the screen getting stuck rotated to portrait. Uninstalling the screen rotation program solved this. And although it does seem to run faster, I think running absolute linux or linux-lite might speed it up more. After all the energy I’d put into getting this far though I don’t have the energy to try other distros again, so Ubuntu it is!

Rollup Plugins for Absolute Beginners

2021-07-26T00:00:00Z

I've recently been getting into the new generation of development tools like wmr, Vite, esbuild and Snowpack. These are causing quite a bit of hype in the community.

So I thought I'd start to get closer to the metal with these tools and how to configure them. Something interesting I found out is that both Vite and wmr have opted to support rollup plugins!

The rollup plugin API is a hell of a lot simpler than writing webpack loaders so this is good news. This also means that understanding rollup plugins and how to write them will continue to be an important skill in the JavaScript world, so let's get into it:

Default Export Function #

A rollup plugin should be a JavaScript module exporting a single function by default. It's also an important convention to start the name with "rollup-plugin-"

At the moment node.js still isn't using esmodules by default, so let's use the ".mjs" extension. Something like this:

// rollup-plugin-mydopeplugin.mjs

export default function myDopePlugin(){}

This function should return an object with some specific keys which rollup is going to look for when running through your code. In the rollup docs these are referred to as "hooks" because they are places you can hook into the build - makes sense!

Name Property #

The most basic and important property to add is the name of your plugin. This will be used by rollup for any error messages or warnings. So let's name our plugin!

// rollup-plugin-mydopeplugin.mjs

export default function myDopePlugin(){
    return {
        name:"my-dope-plugin"
    }
}

Hey this really doesn't seem so bad does it? But we probably want our plugin to do something. At the moment it will do absolutely nothing, but hey, it's bug free!

Transform hook #

This hook is going to let us change our code. It's a function that takes two arguments:

Your code (a string)
The ID of the code (another string)

The string representation of the code is simple enough. But what do I mean by ID? We're JavaScript developers and probably used to IDs being something we use to grab a user or a post when we're processing data. In this case, the ID refers to the string path that was used to import the module. It's the string in the JavaScript import statement after "from" .

So if our source code looked like:

// index.js

import {uppercaseMyString} from './utilities.js'

Then the ID when rollup plugs the utilities import into a transform hook would be "./utilities.js".

What should our transform hook return? There are a few options:

We can return null. This will tell rollup we don't want to transform the code at all and leave it as is.
We can return a string. This will be the code after transformation.
We can return an object. This can give rollup back the transformed code, a sourcemap, something called an abstract syntax tree. It can also provide more information about whether the module has side effects, whether you want to use tree shaking, and specify some special behaviour if another module tries to import a non-existent export.

Let's write our plugin #

Let's write a plugin which will insert a console.log at the start of every JavaScript file that gets imported in our project. We'll start by adding a transform function to our previous code:

// rollup-plugin-mydopeplugin.mjs

export default function myDopePlugin(){
   return {
       name:"my-dope-plugin",
       transform(source,id){

       }
   }
}

Now we'll take a look at the id and make sure we're looking at a JavaScript file. If it doesn't have the ".js" file extension, we'll return null to tell rollup we don't want to do anything with it. So our transform function becomes:

transform(source,id){
    if (id.slice(-3) !== ".js") return null;
}

And finally if it is a JavaScript file, let's concatenate a console.log to the front of our code and return the string.

// rollup-plugin-mydopeplugin.mjs

export default function myDopePlugin(options = {}) {
  return {
    name: "my-dope-plugin",
    transform(source, id) {
      if (id.slice(-3) !== ".js") return null;
      return "console.log('hello from rollup');" + source;
    },
  };
}

And that's our plugin finished!

Now let's use it in a rollup build. Let's npm install rollup and create a "rollup.config.js" file. This file needs a default export which is an object with input and output keys that hold strings to where we want to input and output our source. It also has a key that holds an array of plugins.

We can import our plugin to the rollup config file, then call its function within the plugins array in the rollup config:

// rollup.config.js
import myDopePlugin from './rollup-plugin-mydopeplugin.mjs';

export default {
  input: 'index.js',
  output: {
    file: 'bundle.js',
  },
  plugins: [ myDopePlugin() ]
};

Then when we run our rollup build we just need to tell it to use our config file by adding a --config argument in the command line. If you do a default npm install it'd look something like this:

./node_modules/.bin/rollup --config

And that's it! Hopefully this has made getting your hands dirty in the build step slightly less scary by keeping things super simple.

As mentioned before, you could potentially use this plugin in the build step of a Vite project or a wmr project - not that this particular plugin has any use at all. But this is super empowering that we have some reusability in build systems now.

There's quite a few more rollup hooks to look into if you want to get some more power building these plugins. I'd also recommend checking out This talk about build plugins with Jake Archibald and Jason Miller.

I might try add another blogpost sometime soon getting deeper into rollup plugins as they're so dang useful.

React and Electron

2021-09-02T00:00:00Z

Today I created my first Electron app. It was pretty cool to see something that would usually be stuck within a browser exist as its own unit on the operating system. Getting a "hello-world" going was really easy with the quick-start tutorial.

To get your first electron application built you’ll need to install both:

Electron to get the developer environment going
Electron forge if you want to build/publish the application

Then after you’ve built some of your app you can run:

npx electron-forge import

At the end of this you’ll have 320MB worth of node_modules dependencies in your folder. In my case, forge set itself to use a “squirrel” windows build, which I assume it must have chosen on based on the fact I’m running windows.

After you run the build on the quick start application, you’ll end up with an app 182MB sized application which shows one page, and seems to hog about 80MB of memory when it’s idling away.

These are reasonably big numbers if you’re used to writing native programs in close-to-the-metal languages but the ability to use web technologies to write desktop apps is awesome. And there’s a great article from Niels Leenheer about how Electron applications are fine, and will continue to be useful.

But what about React? #

So I started trying to convert a React single page application into an Electron app. How hard could this be? All I really need to do was convert some JSX to JS right before the app starts right?

So I ended up on this Stack overflow page. The top answer advises us to pull in a babel package called Babel-Register, which looked like a pretty elegant solution. Unfortunately when trying this technique I had all kinds of problems with Electron getting mad at me for using an inline script, and babel yelling at me for using the wrong version even after bumping my dependencies. It wasn’t happening.

So then I took a look at Electron React Boilerplate. This pulled in a mouth-watering amount of tooling. When I ran the command to get the dev server running, the scripts started asking for yarn to be installed. I haven’t got yarn. Maybe I should’ve bitten the bullet and downloaded it but I'm more used to just sticking with npm. I tried a find-and-replace to change all the yarn scripts to npm scripts to see if that would would run - but no dice.

So I threw the Electron React Boilerplate project away and went back to the hello world version. This time I npm installed esbuild - as it’s the speediest, most lightweight way to convert JSX into JS in 2021. Then I set up my npm build and start scripts to run esbuild first, pushing the JavaScript into an intermediate folder, then running the electron build/start command which pulled from the intermediate folder. This worked extremely well. I ran the script and my React app was live! 😈

Here’s a starter for this method of making electron apps with React and esbuild. Go forth and create native apps using your webby skills!

What should “create” do?

2021-09-29T00:00:00Z

We use web apps to make a ridiculous quantity of content every day; videos, blogposts, statuses, documents. We’re ceaselessly clicking “new post”, “upload video”, “start a new document”.

I had an epiphany about what should happen when you click “create” on a web app these days. This epiphany mainly applies when the content takes a long time to create, like writing a document, making a codepen, or, in my app, creating a quiz.

Originally in my quiz app, the “create” link would direct you to a page with a form. You could use this form to build out multi-choice quiz questions then hit “submit” when finished. The new data would get posted to the backend, validated, and added to the database. The backend would then send the data back to confirm the submission was successful. After this, you would have the option to either host the quiz, or further edit the form.

This all sounds pretty sensible. Right?

But the experience I had created felt more like using the web in 2006 than in 2021. In what ways did this user experience transport the user back 15 years?

If you were halfway through the form and refreshed the page or navigated to another page, you would lose everything you had created
It was unclear what the “submit” button would actually do. Would the user be able to keep editing the quiz after it was submitted?
There was no indication to the user suggesting how they might write half the quiz, save, then come back and finish it later

This experience subverted expectations we place on modern web apps. It was tense. It made it feel like you had to get the whole quiz right on the first sitting - like the content was made of delicate china and could break at the smallest push.

We want to be able to take a break and know our work will still be there when we get back. My app used simple CRUD functionality which was similar to what you'd do in old server-rendered applications in the early noughties. We shouldn’t build apps like this any more.

When you’re using google docs, dropbox paper, or codepen and you hit the “create” (or the equivalent button) something different happens. A new space for the item has already been created on the backend by the time you see the page. If you look at the URL for these services, it’s already built out some kind of hashed ID for the content.

None of these services have a “submit” button. A submit button would feel completely out of place. What does submitting mean when I can see it’s been autosaved? The user feels like they’re looking directly at the canonical version of their work. It’s like they’re looking into the database - like the real function of all of these apps is to be a glorified database GUI.

I realise now that my quiz app shouldn’t have separate “create” and “edit” pages. In all the cloud-first content-creation services there is one page to adjust the content whether you are creating it, or you’re coming back to it. In my quiz app when you click the “create” link, it should cause the backend to create a new empty quiz first, then show you the page to edit it.

As you build your quiz, it should be obvious that there is a button that will save your progress and allow you to keep working on it. Or even better, an indication that the quiz is getting autosaved.

I don’t think this is at all a new idea. I think web apps have been using this pattern for years. But I think this little epiphany is highlights one of many details that can decide whether your app will spark anxiety, or spark joy.

Intro to RequestAnimationFrame

2021-12-22T00:00:00Z

requestAnimationFrame is the way to create complex animations with JavaScript. If you’re using an animation library, the chances are that it uses requestAnimationFrame under the hood.

How do you use requestAnimationFrame? We'll start by looking at a simple example to make a line to repeatedly grow:

    <-- HTML -->
    <div id="line"></div>

    /* CSS */
    #line{
    background-color:black;
    height:1px;
    }

    /* JS */
    let width = 0;
    const line = document.getElementById("line");
    let lasttime;
    function step(time) {
      if (lasttime == undefined) {
        lasttime = time;
      }
      let delta = time - lasttime;
      if (width >= 1000) {
        width = 0;
      } else {
        width = width += delta/2;
      }
     line.style.width = `${width}px`;
      lasttime = time;
      window.requestAnimationFrame(step);
    }
    window.requestAnimationFrame(step);

Here’s a Codepen with this code.

There's a lot going on here for such a small animation. 18 lines of pretty weird looking code to make a line get longer.

This is one of the reasons why you might prefer to use CSS for simpler animations. You can actually do the exact same animation with only CSS:

    /* CSS */
    #line{
      height:1px;
      margin-top:10px;
      background-color: black;
      animation-name: line;
      animation-duration: 4s;
      animation-iteration-count: infinite;
      animation-timing-function:linear;
    }
    @keyframes line {
      from {width:0%;}
      to {width:100%;}
    }

And with even less CSS using the animation shorthand property:

    /* CSS */
    #line {
      height:1px;
      margin-top:10px;
      background-color: black;
      animation: line 4s infinite linear;
    }
    @keyframes line {
      from {width:0%;}
      to {width:100%;}
    }

So, I guess the first rule of requestAnimationFrame is ask yourself whether you actually need requestAnimationFrame.

For little animations like this, probably not. But as animations get more complex and interactive, you'll find CSS to be increasingly insufficient. If you need fine-grained control of element positions and timing functions then requestAnimationFrame is the way to go.

To use it, we call the window.requestAnimationFrame function and pass in a callback. Let’s pass in a function that prints “Hello from frame”:

window.requestAnimationFrame(()=>{ console.log("Hello from frame") });

Running this should print “Hello from frame” once. This isn’t particularly exciting. However, what's interesting about it is when the callback function gets run. It looks like it runs immediately, but it actually gets run before the next browser repaint. The browser will usually repaint the page about 60 times a second which is why the callback appears to run instantly.

I'll even prove that it’s not run immediately. When our callback gets called, the browser passes it a decimal number as an argument. This number is the number of milliseconds since the "time origin" of the page (essentially since the page finished loading). We can compare this to the output of performance.now() which prints out the same measurement:

    /* JS */
    console.log(performance.now());
    window.requestAnimationFrame((timestamp)=>{console.log(timestamp)});

When I ran this in codepen it printed off:

    1661.5999999940395
    1680.269

It will print something different every time, depending on browser performance (and a whole bunch of other factors) but there should be a gap of about 30ms between the performance.now() call and the requestAnimationFrame call.

To use window.requestAnimationFrame for animation we can't just throw a callback into it once and leave it. Instead, we want to continously update the page. To get our code to run continuously, we call requestAnimationFrame, passing it our callback function inside the callback. We also need to name our function to do this. We’ll name it step:

    /* JS */
    function step(timestamp){
      console.log(timestamp);
      window.requestAnimationFrame(step);
    }
    window.requestAnimationFrame(step);

This code will continually print timestamps in the console before each browser repaint. But I still wouldn’t exactly call this animation. To actually animate something, we need to change an element’s style on each tick.

Let’s add a centered red box to the page:

    <-- HTML -->
    <div id="growing-box"></div>

    /* CSS */
    body{
      height:100vh;
      margin:0;
      display:flex;
      justify-content:center;
      align-items:center;
    }
    #growing-box{
      width:40vmin;
      height:40vmin;
      background-color:#af0000 ;
    }

Now let’s use requestAnimationFrame to make the box slowly grow. The best way to do this is using the transform CSS property, and the scale function, as this gets GPU accelerated by the browser. So, we grab the box , set a scale variable to start with a value of 1, add 0.01 to the scale on each iteration, and apply the scale to the element on each iteration:

    /* JS */
    let square = document.getElementById("growing-box");
    let currentScale = 1;
    function step(timestamp){
      currentScale += 0.01;

      square.style.transform = `scale(${currentScale})`

      window.requestAnimationFrame(step);
    }
    window.requestAnimationFrame(step);

Here’s a codepen showing what this looks like.

Now we have a growing square, but there’s a couple of problems.

First, it will grow forever. We need to add some kind of check to make sure that it stops (or repeats) at some point.

Let’s add a check which will test whether the scale is equal or above 2, and, if so, reset it to 1.

Second, it will grow unevenly. This is because the interval between browser repaints varies, but we’re changing the scale the same amount each time. It won’t be very noticeable on faster devices, but will become a problem as repaints get more intermittent.

Let’s measure the time between the last frame and the current frame (we’ll use the convention of naming this delta). We use the delta to figure out how far we need to grow the square. We’ll divide delta by 30, just to make sure the square isn’t growing too fast.

    /* JS */
    let square = document.getElementById("growing-box");
    let currentScale = 1;
    let lastTime;

    function step(timestamp){

      if(lastTime==undefined){
        lastTime = timestamp;
      }

      let delta = timestamp - lastTime;
      currentScale += 0.01 * (delta / 30);

      if(currentScale>=2){
        currentScale= 1;
      }

      square.style.transform = `scale(${currentScale})`

      window.requestAnimationFrame(step);
      lastTime= timestamp;
    }
    window.requestAnimationFrame(step);

Now we have a box which repeatedly grows then jumps back down to being small again.

We can still do better.

At the moment, there isn’t any easy way to change the speed of the animation, how big the square gets, or whether the animation loops. So let’s make some variables.

    let animationLength = 1;
    let animationLengthMillis = animationLength *1000;
    let timeThrough = 0;
    let startScale = 1;
    let endScale = 2;
    let looping = true;

We’re storing the animation length (in both seconds and milliseconds), the amount of time that has passed through the current iteration, the start/end points for our scale property, and finally whether the animation should loop.

We need to change a couple things to use these. Firstly, we can use the timeThrough variable to keep track of how far through the animation we are, and add our delta variable to this on each tick.

Then we can get a fraction value of how far through the animation we are by dividing the timeThrough by the animationLengthMillis. Then we set the currentScale to the startScale plus the difference between endScale and startScale multiplied by the fraction through the animation we are:

    let percentThrough = timeThrough / animationLengthMillis;
    let currentScale = startScale + (endScale - startScale) * percentThrough;

This is a lot. But basically what it’s doing is setting the scale based on where we are in the animation.

Now we need to make sure the animation lasts the correct length. So, we check to see if the time that has passed is more than or equal the animation length. If so, we set timeThrough back to zero or return to end the animation (depending on whether we're looping or not).

Here’s how it looks:

    /* JS */
    let square = document.getElementById("growing-box");
    let animationLength = 1;
    let animationLengthMillis = animationLength * 1000;

    let lastTime;
    let timeThrough = 0;
    let startScale = 1;
    let endScale = 2;
    let looping = true;

    function step(timestamp) {

      if (lastTime === undefined) {
        lastTime = timestamp;
      }
      let delta = timestamp - lastTime;

      timeThrough += delta;

      if (timeThrough >= animationLengthMillis) {
        if (looping) {
          timeThrough = 0;
        } else {
          return;
        }
      }

      let percentThrough = timeThrough / animationLengthMillis;
      let currentScale = startScale + (endScale - startScale) * percentThrough;

      square.style.transform = `scale(${currentScale})`;

      window.requestAnimationFrame(step);
      lastTime = timestamp;
    }
    window.requestAnimationFrame(step);

And, here’s the codepen.

You can see how you can edit these variables to your taste to make the animation look the way you need.

Animation can make user interfaces less jarring, and more human-friendly. They’re a necessary tool to have in your user-interface toolbox. For something a bit less trivial you can check out a minimal notetaking app with some squishy animations over at this codepen, and for more information on requestAnimationFrame you can’t go wrong with the MDN web docs.

Explorations in C, C++, Win32, and SDL

2022-01-10T00:00:00Z

lately I've been experimenting with C, C++, Win32, and SDL. The first thing I've noticed is we don't know how good we have it in the JavaScript/web development world:

Blog posts and online resources #

You can find great documentation for web stuff everywhere. JavaScript has blown up in a time when all kinds of people are becoming coders. There are a thousand accessible blog posts for any topic. And these posts actually get surfaced by Google!

C++ was blowing up when at a time when developers were a small subsection of society. If you were a coder, you had to be passionate. Of course you were - you were making command line apps! And if you were lucky enough to build GUIs, you had a lot of patience. Blog posts about C and C++ are relatively difficult to find and usually assume a lot of previous knowledge. They usually reference star wars. They argue about the differences between C and C++ and their use cases. They talk about how pointer arithmetic separates the men from the boys. 🤮

But it makes sense that we’re going to get more resources about the languages of the web on the web. A lot of the knowledge on C and C++ exists in those old things called books!

Language features #

We also have a lot of friendly language features in JS. JavaScript arrays aren't real arrays. They're more like cuddly rows of boxes that will pat you on the head and buy you an ice-cream for accessing an index.

In JavaScript we have problems with differing module types from the migration from common.js to esm, but we have nothing compared to the nightmare of header file includes, DLLs, and Makefiles. Also in C++ you need to reimplement rounding floating point numbers into ints - There is no comfy Math class for you here.

Win32 and Microsoft #

Navigating Microsoft documentation remains extremely difficult. Lists of links to explain different APIs take you on a rocky journey. The win32 API is obtuse and ugly. It demands a huge number of arguments in functions to get a window showing. You need gigantic event switch statements to take user input.

Why though? #

This all said, I think all of this is worth taking a look at. Why? Because C, C++ and Win32 create efficient programs. In both program size and runtime speed they beat electron apps. In a lot of ways they are better for the user. They're close to the metal and teach you about what the computer is actually doing. Our browsers are written with C++, our youtube videos are processed with C++, our databases are written with C++. And all the tools for C and C++ are free for the taking on the internet. Download Mingw on windows and open up your favourite text editor and you’re ready to start compiling! There are more steps here than making an index.html file, but not that many more steps.

I know that if you want a systems language for cool kids, there’s rust. I know that the majority of web apps will be fast enough with JavaScript and PHP based tools (and all the popular alternatives). I know that the high learning curve of C and C++ mean they’re probably not the most bang-for-your-buck languages for a web developer to learn in early 2022. But I’m always obsessed with the old ways of doing things and can’t help myself exploring.

SDL #

SDL is a toolkit for C and C++ to create “multimedia applications” (games. It’s for games.). Lazyfoo has a great tutorial on how to use it. If you’ve made a few command line programs and want to try making games, I’d recommend trying SDL. It’s so lightweight, and gives you a nice way to dig into building a little game engine and see what C and C++ are all about. It’s cross platform, has a few more resources around online, and is a lot nicer than trying to write raw openGL code yourself.

More thoughts #

Do I even know what pointers are? The way I think about them is indexes of a giant hidden array which directly exposes the computers memory. I don’t know if this is correct, but I almost feel like a real programmer hitting that star key *.

And could I find a way to hook into and extend node with C++? Looks like I could! Assuming I have the time and willingness. And could I ever finish a game with C++? Probably not if I want to have money and/or time.

My JavaFX ListViews are doubling up

2022-01-18T00:00:00Z

You’re making a JavaFX app and you want to render a collection properly by using a ListView or TableView. These components are great: you point them to an ObservableList and all you need to do is update this list and the views will automatically be updated. You can also generate a bunch of your favourite JavaFX UI elements for each item in the list — including buttons, labels, inputs — any components at all!

There’s a problem you might run into though. We’ll see it soon.

Hello ListView #

We’ll start with the “hello world” of JavaFX list views. The code below is essentially what you’d find in the JavaFX ListView docs. But I’ve rendered inside of an application scene:


    import javafx.application.Application;
    import javafx.collections.FXCollections;
    import javafx.collections.ObservableList;
    import javafx.scene.Scene;
    import javafx.scene.control.ListView;
    import javafx.scene.layout.VBox;
    import javafx.stage.Stage;

    public class Main extends Application {

            @Override
            public void start(Stage primaryStage) throws Exception {

                     ObservableList<String> names = FXCollections.observableArrayList(
                              "Julia", "Ian", "Sue", "Matthew", "Hannah", "Stephan", "Denise");
                     ListView<String> listView = new ListView<String>(names);

                    VBox pane = new VBox();
                    pane.getChildren().add(listView);
                    Scene scene = new Scene(pane, 700, 500);
                    primaryStage.setScene(scene);

                    primaryStage.show();
            }
    public static void main(String[] args) {
            launch(args);
    }
}

This works fine and prints off a nice list of names.

Cell factories #

What if we wanted to customize our ListView?

Let’s make it render the following for each item in our observable list:

A Text component for the name
A Region component to separate the name from the others
A TextField component to edit the name
A submit Button to confirm the change

We’ll wrap these in an HBox component so they’re nicely displayed horizontally on the screen.

You can do this using a cell factory - which needs to be a class that implements the JavaFX CallBack interface. In this class we override the call method to return a new ListCell object.

In the ListCell object we return, we want to override the updateItem method and call setGraphic inside. In the arguments of setGraphic, we pass the component that should be created for each cell. We’re going to pass it our HBox after we’ve created it and filled it with the list of components from before.

I promise it’s not too bad. Here’s what it looks like:


    /* Main.java
    Do not write your JavaFX like this - we need to fix this up!
    */
    import javafx.application.Application;
    import javafx.collections.FXCollections;
    import javafx.collections.ObservableList;
    import javafx.geometry.Pos;
    import javafx.scene.Scene;
    import javafx.scene.control.Button;
    import javafx.scene.control.ListCell;
    import javafx.scene.control.ListView;
    import javafx.scene.control.TextField;
    import javafx.scene.layout.HBox;
    import javafx.scene.layout.Priority;
    import javafx.scene.layout.Region;
    import javafx.scene.layout.VBox;
    import javafx.scene.text.Text;
    import javafx.stage.Stage;
    import javafx.util.Callback;

    public class Main extends Application {

            @Override
            public void start(Stage primaryStage) throws Exception {

                     ObservableList<String> names = FXCollections.observableArrayList(
                              "Julia", "Ian", "Sue", "Matthew", "Hannah", "Stephan", "Denise");
                     ListView<String> listView = new ListView<String>(names);

                     class PersonCellFactory implements Callback<ListView<String>, ListCell<String>> {

                                    @Override
                                    public ListCell<String> call(ListView<String> listView) {
                                               return new ListCell<String>(){
                                                @Override
                                                public void updateItem(String nameString, boolean empty) {
                                                    super.updateItem(nameString, empty);
                                                    Text nameText = new Text(nameString);
                                                    Region region1 = new Region();
                                                    TextField nameField = new TextField(nameString);
                                                    Button changeNameButton = new Button("update");
                                                    HBox box = new HBox();
                                                    HBox.setHgrow(region1, Priority.ALWAYS);
                                                    box.setAlignment(Pos.BASELINE_LEFT);
                                                    box.getChildren().addAll(nameText,region1,nameField,changeNameButton);
                                                    setGraphic(box);
                                                }
                                            };
                                    }
                            }
                    listView.setCellFactory(new PersonCellFactory());
                    VBox pane = new VBox();
                    pane.getChildren().add(listView);
                    Scene scene = new Scene(pane, 700, 500);
                    primaryStage.setScene(scene);

                    primaryStage.show();
            }
    public static void main(String[] args) {
            launch(args);
    }
}

If you run this you’ll get something a little weird, where you get a clickable list of names, then a big list of items underneath in which the button and texfield is clickable, but not the rest. This is sort of what we want, but not really.

Checking for empty cells #

What we need to do is add a check in updateItem to make sure that if it’s an empty cell, we don’t add anything in. This is what you might think this should like:


    @Override
    public void updateItem(String nameString, boolean empty) {

            super.updateItem(nameString, empty);
            /*
            * Uh oh! This is buggy! Don't worry we'll fix it soon
            */
            if (empty) {
                    return;
            }
            Text nameText = new Text(nameString);
            Region region1 = new Region();
            TextField nameField = new TextField(nameString);
            Button changeNameButton = new Button("update");
            HBox box = new HBox();
            HBox.setHgrow(region1, Priority.ALWAYS);
            box.setAlignment(Pos.BASELINE_LEFT);
            box.getChildren().addAll(nameText, region1, nameField, changeNameButton);
            setGraphic(box);
    }

Okay, when we run it it seems okay? We’re only rendering the items that aren’t empty - just what we wanted.

But let’s say that we add a button afterwards to add a new person to our list:


    listView.setCellFactory(new PersonCellFactory());
    /* New code from here */
    Button newPersonButton = new Button("Add person");
    newPersonButton.setOnAction(event->{
            names.add("Johnny");
    });
    VBox pane = new VBox();
    pane.getChildren().add(listView,newPersonButton);
    /* New code ends here */
    Scene scene = new Scene(pane, 700, 500);
    /* etc */

When we add this, everything looks fine to start with, but when we click the newPersonButton we run into our problem. The new item, “Johnny”, gets added, but all the other items seem to appear under it in reverse order! What is going on?

Why isn’t my JavaFX ListView working? #

This is something to do with the internals of the way the ListView interacts with cell factories. What we need to do when the cell is empty is not just return, but set the graphic as null.

This is because the updateItem method is used not only to create the cell, but also to clean it up!

This is how our check should look:

if (empty) {
/* This is the important part */
setGraphic(null);
return;
}

And here’s how the whole program should look at this point:

    /* Main.java
    Do not write your JavaFX like this - we need to fix this up!
    */

    import javafx.application.Application;
    import javafx.collections.FXCollections;
    import javafx.collections.ObservableList;
    import javafx.geometry.Pos;
    import javafx.scene.Scene;
    import javafx.scene.control.Button;
    import javafx.scene.control.ListCell;
    import javafx.scene.control.ListView;
    import javafx.scene.control.TextField;
    import javafx.scene.layout.HBox;
    import javafx.scene.layout.Priority;
    import javafx.scene.layout.Region;
    import javafx.scene.layout.VBox;
    import javafx.scene.text.Text;
    import javafx.stage.Stage;
    import javafx.util.Callback;

    public class Main extends Application {

            @Override
            public void start(Stage primaryStage) throws Exception {

                    ObservableList<String> names = FXCollections.observableArrayList("Julia", "Ian", "Sue", "Matthew", "Hannah",
                                    "Stephan", "Denise");
                    ListView<String> listView = new ListView<String>(names);

                    class PersonCellFactory implements Callback<ListView<String>, ListCell<String>> {

                            @Override
                            public ListCell<String> call(ListView<String> listView) {
                                    return new ListCell<String>() {
                                            @Override
                                            public void updateItem(String nameString, boolean empty) {

                                                    super.updateItem(nameString, empty);
                                                    if (empty ) {
                                                            /* This is the important part */
                                                            setGraphic(null);
                                                            return;
                                                    }
                                                    Text nameText = new Text(nameString);
                                                    Region region1 = new Region();
                                                    TextField nameField = new TextField(nameString);
                                                    Button changeNameButton = new Button("update");
                                                    HBox box = new HBox();
                                                    HBox.setHgrow(region1, Priority.ALWAYS);
                                                    box.setAlignment(Pos.BASELINE_LEFT);
                                                    box.getChildren().addAll(nameText, region1, nameField, changeNameButton);
                                                    setGraphic(box);
                                            }
                                    };
                            }
                    }

                    listView.setCellFactory(new PersonCellFactory());

                    Button newPersonButton = new Button("Add person");
                    newPersonButton.setOnAction(event -> {
                            names.add("Johnny");
                    });
                    VBox pane = new VBox();
                    pane.getChildren().addAll(listView, newPersonButton);
                    Scene scene = new Scene(pane, 700, 500);
                    primaryStage.setScene(scene);

                    primaryStage.show();
            }

            public static void main(String[] args) {
                    launch(args);
            }
    }

But wait! This still isn't ideal. There's a better way!!!!

We're doing too much work in our updateItem method! A big thank you to Dirk Lemmermann for pointing this out.

Reducing the amount of work in the updateItem method #

It turns out the updateItem method gets called all the time. This includes getting called in response to select and focus events as well as changes in the data model. At the moment our code is rebuilding the box and all of its child components from scratch for every item, every single time one of these events happens.

To reduce the amount of work that we're doing in our updateItem method, we can build the child components once in the cell factory (or if you make a custom cell class: in the constructor). After that, the only thing our updateItem method needs to do, is call setText on the nameText and nameField components.

To stop the cell from showing up when it's empty we can use a superpower JavaFX provides: Binding. We want to bind the visibility property of the HBox to the inverse of the empty property of the ListCell. Here's what this looks like:

box.visibleProperty().bind(cell.emptyProperty().not());

And here's how it looks in context:


import javafx.application.Application;
import javafx.collections.FXCollections;
import javafx.collections.ObservableList;
import javafx.geometry.Pos;
import javafx.scene.Scene;
import javafx.scene.control.Button;
import javafx.scene.control.ListCell;
import javafx.scene.control.ListView;
import javafx.scene.control.TextField;
import javafx.scene.layout.HBox;
import javafx.scene.layout.Priority;
import javafx.scene.layout.Region;
import javafx.scene.layout.VBox;
import javafx.scene.text.Text;
import javafx.stage.Stage;
import javafx.util.Callback;

public class Main extends Application {

	@Override
	public void start(Stage primaryStage) throws Exception {

		ObservableList<String> names = FXCollections.observableArrayList("Julia", "Ian", "Sue", "Matthew", "Hannah",
				"Stephan", "Denise");
		ListView<String> listView = new ListView<String>(names);

		class PersonCellFactory implements Callback<ListView<String>, ListCell<String>> {

			@Override
			public ListCell<String> call(ListView<String> listView) {

				Text nameText = new Text();
				Region region1 = new Region();
				TextField nameField = new TextField();
				Button changeNameButton = new Button("update");
				HBox.setHgrow(region1, Priority.ALWAYS);
				HBox box = new HBox();
				box.setAlignment(Pos.BASELINE_LEFT);
				box.getChildren().addAll(nameText, region1, nameField, changeNameButton);

				ListCell<String> cell = new ListCell<String>() {
					public void updateItem(String nameString, boolean empty) {
						super.updateItem(nameString, empty);
						nameText.setText(nameString);
						nameField.setText(nameString);
					}
				};
				cell.setGraphic(box);
				box.visibleProperty().bind(cell.emptyProperty().not());
				return cell;
			}
		}

		listView.setCellFactory(new PersonCellFactory());

		Button newPersonButton = new Button("Add person");
		newPersonButton.setOnAction(event -> {
			names.add("Johnny");
		});
		VBox pane = new VBox();
		pane.getChildren().addAll(listView, newPersonButton);
		Scene scene = new Scene(pane, 700, 500);
		primaryStage.setScene(scene);

		primaryStage.show();
	}

	public static void main(String[] args) {
		launch(args);
	}
}

Nice! Custom JavaFX ListViews behaving just the way we want them to and not doing to much work to recreate the UI. 😌

Now onto writing the code to edit the items. . .

The callbacks are out to get you

2022-03-03T00:00:00Z

I remember some of the first lines of JavaScript code I looked at were in Jon Duckett’s JavaScript & JQuery. This book and HTML & CSS are the canonical get-into-web-development books. They are truly singular in that they’re the only ever textbooks to actually LOOK GOOD.

Anyway, I remember coming to the section of the book about adding click handlers. There was a code sample that looked something like this:

    let button = document.getElementById("button");
    button.addEventListener("click", onClick);

After this it defined the onClick function:

    function onClick(event){
        console.log("Button has been clicked");
    }

To anyone who knows and loves JavaScript, this is a very simple bit of code.

But it’s also something I remember finding confusing. And since I read that book back in 2019 I had forgotten how confusing it was, until I witnessed some programmers who were new to JavaScript getting caught by it.

My problem was with the arguments to the addEventListener method. This method takes in two arguments. The first is a string describing which event you want to respond to - this makes sense. But what is the second argument?

Well it’s a function. It’s describes what will happen after the button is clicked.

Okay, it’s a function. We write those with round brackets at the end. So I should add the function to the event listener like this, right?

     button.addEventListener("click", onClick());

WRONG! If you write your click handler like this, you’ll click your button and it confusingly won’t do anything. Even though onClick is a function, we have to remove the round brackets after onClick for this line of code.

Why are these round brackets so evil you ask? Every time you’ve seen a function in JavaScript up until now, it’s had a nice pair of round brackets afterward. For instance, when I declare a function:


    // Has some round brackets with no arguments
    function myFunction(){
        console.log("I am such a great function");
    }
    // Has some round brackets with arguments too!
    function add(num1, num2){
        return num1 + num2;
    }

And when I call a function:

    // Has some round brackets when we call them
    myFunction();
    add(3,4);

But all of a sudden we’re doing something that involves functions, but we’re not placing curly brackets after its name? Why?

Functions as a first class citizen #

The reason for what is going on here is that JavaScript treats functions as a first class citizen. Functions are allowed to do anything that any variable can do - get passed into other functions, get pointed to by multiple variables, get mutated. All sorts.

This means this syntax:

    function myFunction(){
        console.log("I am such a great function");
    }

Does pretty much the same thing as this syntax:

    let myFunction = function(){
        console.log("I am such a great function");
    }

Both are valid JavaScript syntax, and both create a variable called myFunction which points to a function. We can pass this variable (which points to a function) INTO ANOTHER function:

    // Declare first function
    function myFunction(){
        console.log("I am such a great function");
    }
    // Declare second function
    function anotherFunction(input){
        input();
    }
    // Here we pass the first function into the second one
    anotherFunction(myFunction);

The big difference when we put the round brackets after a function, is that this means the function will get excecuted immediately when the JavaScript is executed. It’s a way of saying “run this now!”, and if there are arguments “run this now with these arguments”.

When the JavaScript is executing and gets to that line, it will run the function and replace it with its result.

So when we do this:

    button.addEventListener("click", onClick());
    function onClick(event){
        console.log("Button has been clicked");
    }

We’re passing the event listener what our onClick event returns.

And it doesn’t return anything. So, what we’re actually passing into the event listener is undefined.

Instead of passing in undefined, we want to pass our function. That’s why we don’t have a pair of round brackets after the second argument.

It’s a way of saying, “hey, can you do this thing later?”

And that, is a callback.

Why is SQL weird?

2022-05-10T00:00:00Z

If you make a search for "SQL” on Seek, you’ll find thousands of jobs. IBM developed SQL in the early 70s and won the war of relational database management languages. There are millions of SQL databases running in data centres around the world. It’s an important tool to learn.

Moving logic out of the application layer and into SQL can improve performance and make systems easier to understand. SQL gets used by business analysts, data scientists and other non-developers to access vaults of information that would otherwise be hidden. Some trendy Node.js apps avoid SQL by using MongoDB, but even the node ecosystem sometimes has to choose boring technology.

SQL is a weird language if you’re used to C-like languages. For a many of us it will be one of the first domain-specific languages we learn. It’s completely different in syntax, scope, and mindset from imperative and object-oriented programming languages. In some ways it feels more similar to functional programming, as you need to change your thinking from step-by-step to declarative.

SQL Order of Operations #

The first thing I found strange in SQL was comparing the order of operations to the order you write an instruction with. In a C-like language, we’re used to things going from top to bottom: A series of instructions ending in semicolons that start at the top and finish at the bottom. Within lines, orders of operations can get a little more complicated, however, I’ve always found C-like languages intuitive. A lot of the order matches intermediate-school algebra we’re mostly familiar with.

But in SQL, the order of operations are more idiosyncratic. Look at this query:

SELECT [name],[breed],[age] FROM [dbo].[dogs] WHERE [age] > 5 ORDER BY [name];

SQL operations follow this order:

FROM
JOIN
WHERE
GROUP BY
HAVING
SELECT
ORDER BY

This means in the previous query, operation starts from the middle of the line (FROM), then moves right to WHERE, then jumps back to the start of the line with SELECT then bounces back over to the end of the line with ORDER BY.

The other strange thing is that you can’t change the order you write instructions. You can't start with order by then write FROM. You have to follow the correct order to write the instructions, and it’s a different order from the execution order?

I find this quite stressful to keep in my head, but an SQL professional would probably argue that it’s a human-friendly way to phrase the language. "Plus!" They scream from the rooftops. "It’s declarative! You shouldn’t have to worry about the order of operations anyway! Let the engine take care of it!"

The Dunning Kruger query language #

Learning introductory SQL is easy. There are a lot of introductory courses that will tell you the simple stuff: SQL databases are made up of tables, which have typed columns and hold a number of rows in which you store data, and you can hook up different tables up with foreign keys and joins to define relationships.

But as soon as you get into intermediate SQL the difficulty curve steepens. This programming language was made to give you Dunning Kruger syndrome. “I know how to create a table with the columns I need, and do CRUD operations on it!” you’ll excitedly say.

But then you find out about about Views, Subqueries, Stored Procedures, indexing, coalescing, user-defined functions, not to mention the biggest topic of them all: performance. You fall off a complexity cliff into an “I know nothing” zone. This might happen in a lot of languages, but most resources seem to tell you "well done, you know SQL well enough to be a developer" after teaching a few surface-level concepts.

Meta-SQL #

The thing about SQL is you do everything with SQL. There are a lot of different GUI apps that allow you to look into a database’s structure and data without running SQL queries. But the more you use these apps to gather information about your database, the more you realise that you’re doing it wrong! What’s the best way to get a list of all the table names? Don’t try and use your GUI client. Instead, run this query:

SELECT TABLE_NAME FROM INFORMATION_SCHEMA.TABLES;

What’s the best way of searching for a view you made? Don’t use the GUI! Run this:

SELECT * FROM sys.views WHERE name like '%MyView%';

What’s the best way to see how many records are in a table? It’s this:

SELECT count(*) FROM [dbo].[dogs];

Okay maybe that’s not the best way to find the number of records in a table - but it’s good for most cases.

SQL isn't only for accessing, deleting, and changing data. It's also for describing data and building databases. And sometimes it feels weird to do both of these using the same quote unquote “domain specific language”, as they're actually quite different tasks.

You can set database table names based on data in another table. You can store SQL configuration variables in the same database that you're configuring. You can use SQL to drive other SQL. That's weird.

Injection #

One of my first blog posts on here was about how to avoid SQL injection using PHP data objects. We would never have needed this sort of thing if we didn’t have to mix SQL into other programming languages. The fact that developers use strings to build out executable queries puts us all in danger, as the Computerphile YouTube channel shows us in this vid.

Who knows how many apps are running right now with an opening for us to type drop table [dbo].[users]; and watch the world burn. Who knows how many passwords have been claimed. Well, Hold Security made an estimate that at least 1.2 billion unique email-password pairs had been stolen using SQL injection. That was 7 years ago now, so hopefully everyone’s moved on to MongoDB since then. Problem solved.

Part of the reason SQL feels weird is that we're always calling it from another language. Plus we're always scared of the potential repercussions of doing this incorrectly.

Maybe I'm the weird one #

I haven't spent as much time with SQL as I have with HTML, JavaScript, CSS, PHP, or Java. This could be the real reason I'm asking why it's weird. Not because of any flaw with the language, but with me.

There you go. The real domain-specific-database-management-language was the friends we made along the way.

Compile your own Apache Module in C

2022-06-16T00:00:00Z

Before we jump in, I should let you know these instructions are going to be for 64 bit Windows only. I'll be compiling on Windows and for Windows. If you're running Mac or Linux I'm afraid you'll have to find another (probably easier) tutorial.

Compiling Apache for Windows is probably a pretty cooked thing to do anyway, as you'll usually want to deploy Apache on a nice cheap linux virtual machine. The good thing is the source code should be cross platform. The Apache Portable Runtime allows for this. But the compilation steps and downloads won't be. And I haven't double-checked that the source does compile properly on Mac or Linux.

The Apache Server #

The Apache Server was the best way to get a website running for most of the lifetime of the web. As of 2022 it holds a 31.4% market share of web server statistics. It continues to run a large part of the internet, in no small part due to Apache being a first class way to deploy a PHP application (it hosts a whole lot of Wordpress). Apache is the A in LAMP stack (Linux, Apache, MySQL and PHP). It's the stack that ruled the naughties, for better or for worse. Even though it has had its market share cannibalised by Nginx over the last decade, the Apache Server remains an important piece of tech.

How does Apache connect to PHP? One way is to run PHP is with FastCGI, a cross-platform way to get servers to talk to other programs on your system. More likely though, you'll be running it as an apache module.

The Apache server is a modular system, where modules execute logic across different parts of the request/response cycle of the HTTP protocol. These modules are .SO files - shared library files. They are binaries that live in Apache's modules folder. Apache has the ability to call functions they expose. You can filter which modules apply to which requests and configure how they work in with Apache's config file, and also . Hidden deep within the Apache documentation is a guide for Developing modules. We'll be half-following this guide.

Getting Apache for Windows #

First we'll need to download Apache. You'd expect to download Apache from the Apache website, but they only hold the source code and Unix binaries. Instead, you can grab Apache for Windows from third party vendors such as Apache Lounge. Download the Windows 64 version, and extract it into your favourite folder. The folder structure inside Apache should look like this:

modules - Where all the different modules get stored. There will already be many modules provided by Apache already there
bin - Where the Apache binaries live. This is where we go to start Apache
cgi-bin - Where you'd store binaries for the Common Gateway Interface. This is the way PHP used to be handled but it's less efficient than the techniques described earlier
conf - Configuration files
error - Error messages in different languages
htdocs - The public files you want to expose to the internet - eg. where you put your index.html
icons - Icons
include - Header files we include in C code when writing our module
lib - Library files we link to our C object files when compiling our module
logs - Logs - a good place to look when things aren't working
manual - A folder with a single text file which is just a link to the Apache website. Yup.

To get a server running, we first set Apache up as a Windows Service. Open up Powershell as administrator and navigate to Apache's bin folder, then run this command:

./httpd.exe -k install

Now you have it installed as a service. This makes it a little but easier to run. We can start our server with this command:

NET START Apache2.4

After running this, you should see a message to say the server is starting. Head to your browser and type in "localhost" in the address bar. You should see a big H1 in times-new-roman saying "It works!". You can stop Apache by heading back to Powershell and running this command:

NET STOP Apache2.4

It might take a few seconds to run its cleanup.

Getting Visual Studio Build Tools #

We're going to write some C code, but we need a way to turn it into a binary shared library. If only there was some way to do that. Oh wait, that's what a compiler does. Let's grab a command-line version of Microsoft's MSVC compiler. Head to the Visual Studio Downloads Page, and scroll past all the IDEs (grab Visual Studio Code if you don't already have it - it's a great editor).

Under "All downloads" click into "Tools for Visual Studio". Scroll to the bottom and download "Build tools for Visual Studio". This is going to get us a compiler we can use from the command line.

But first we need to go through the Visual Studio installer which you can install and open. Select "Desktop development with C++" on the left. On the right, make sure the boxes are ticked for "MSVC" and "Windows 10 SDK". Install these.

Now a good idea is to open up "x64 Native Tools Command Prompt for VS 2022", and type in the letters:

cl

Then hit return and it should print off the version of the Windows Visual Studio C compiler you have installed.

That's the installing part done! Now let's write some code for our Apache module.

Writing the boilerplate #

We're going to make a module which will respond to any request with "hello apache". We'll be writing this in C. Create a file called "mod_hello_apache.c". The first thing we need to do is include headers from the Apache libraries we're going to use:

#include <httpd.h>
#include <http_protocol.h>
#include <http_config.h>

These libraries are provided by Apache for developers to build modules. They give us cross-platform functions to send data to respond to HTTP requests. We also use them to tell Apache how to call our module.

Now we need to expose a module struct. On start-up, Apache will look for this struct and pass it configuration data. It will also call the last item in the struct - a function which we use to set our module up:

module AP_MODULE_DECLARE_DATA hello_apache_module =
{
    STANDARD20_MODULE_STUFF,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
    hello_apache_hooks   /* Our hook registering function */
};

In our code the set up function is called hello_apache_hooks. We don't really need any configuration for this simple module, so we're just going to leave the configuration arguments (the middle 5 arguments) as null. Why does the name of our set-up function include the word "hooks"? It's because we use this function to tell Apache how our module will "hook" into the request/response cycle. We use hooks to tell Apache what happens when.

Let's write our hello_apache_hooks function now:

static void hello_apache_hooks(apr_pool_t *pool)
{
    ap_hook_handler(hello_apache_handler, NULL, NULL, APR_HOOK_MIDDLE);
}

What is being passed into the hello_apache_hooks function? A pointer to something called a pool. For each request that hits our module, Apache will create a pool, which is an object to help us manage memory. Using a pool we can fill up memory with data for our handler, and Apache will clean everything up at the end of the processing cycle (when we call a cleanup function). This prevents memory leaks.

We've all heard about how scary C can be with its lack of guard rails for memory allocation. Pools make it a little safer.

We don't really need to do any complex memory assignment so we leave the pool alone.

Then we call an Apache function called ap_hook_handler. This registers that we want to run a function called hello_apache_handler in the middle of the request/response cycle. There are multiple points in the cycle we can hook into, including first, last, and middle.

We've put our handler in the middle, but you can imagine we might have a security module which kicks out blacklisted IP addresses before they reach our handler, or a module which attaches cache control headers to outgoing images. The middle two arguments are for telling Apache about functions to run before and after this one. We'll leave these as null.

Writing the handler function #

Okay now we're finished with the boilerplate! Well done, for making it this far!

Let's write our hello_apache_handler. This is a static function that contains the logic we run when a request comes in. We'll take in a pointer to a request_rec as a parameter. This is a struct which will describe all the information about the HTTP request - all the request headers, any query parameters, and a whole lot of other information you might need.

Our function will return an integer which serves as a status code to tell Apache whether our handler was successful.

static int hello_apache_handler(request_rec *r){}

Now let's do some checks. First we want to make sure that this request is actually meant to be for our handler (it's our job as module developers to do this):

static int hello_apache_handler(request_rec *r)
{
    if (!r->handler || (strcmp(r->handler, "hello_apache") != 0))
    {
        return DECLINED;
    }
}

Here we're making sure that the request actually needs a handler, then we make sure that the handler is our module.

We need to make sure this is a GET request. This is going to be the request method our module will respond to:

static int hello_apache_handler(request_rec *r)
{
    if (!r->handler || (strcmp(r->handler, "hello_apache") != 0))
    {
        return DECLINED;
    }
     if (r->method_number != M_GET)
    {
        return HTTP_METHOD_NOT_ALLOWED;
    }
}

If it's not a GET request, we return a status code provided by the Apache libraries to say that this method isn't allowed.

And now let's send something to the client. First we'll send a response header to say this is HTML, then we'll send some HTML over the wire:

static int hello_apache_handler(request_rec *r)
{
    if (!r->handler || (strcmp(r->handler, "hello_apache") != 0))
    {
        return DECLINED;
    }
    if (r->method_number != M_GET)
    {
        return HTTP_METHOD_NOT_ALLOWED;
    }
    ap_set_content_type(r, "text/html;charset=utf8");
    ap_rputs("<html>"
             "<head><title>Hello apache</title></head>"
             "<body>"
             "<h1>Hello from apache module!</h1>"
             "</body>"
             "</html>",
             r);
    return OK;
}

We return OK at the end of the function to say things went smoothly. Nice.

The entire module source #

Here's the entire module all together:

#include <httpd.h>
#include <http_protocol.h>
#include <http_config.h>

static int hello_apache_handler(request_rec *r)
{
    if (!r->handler || (strcmp(r->handler, "hello_apache") != 0))
    {
        return DECLINED;
    }
    if (r->method_number != M_GET)
    {
        return HTTP_METHOD_NOT_ALLOWED;
    }
    ap_set_content_type(r, "text/html;charset=utf8");
    ap_rputs("<html>"
             "<head><title>Hello apache</title></head>"
             "<body>"
             "<h1>Hello from apache module!</h1>"
             "</body>"
             "</html>",
             r);
    return OK;
}

static void hello_apache_hooks(apr_pool_t *pool)
{
    ap_hook_handler(hello_apache_handler, NULL, NULL, APR_HOOK_MIDDLE);
}

module AP_MODULE_DECLARE_DATA hello_apache = {
    STANDARD20_MODULE_STUFF,
    NULL,
    NULL,
    NULL,
    NULL,
    NULL,
    hello_apache_hooks};

That's our module! The functions need to be defined in this order to make sure that they've been declared before they get referenced.

Compiling the module #

Apache provides a tool called APSX. This is a perl script that looks at your machine and the file you're compiling and sets the appropriate compiler flags. I gave it a try but I couldn't get it working, so instead we're going to just use the the Visual Studio C compiler we downloaded earlier - MSVC.

Before you use MSVC, you'd usually need to set a bunch of environment variables for where windows libraries and headers live. An easier way is to open "x64 Native Tools Command Prompt for VS 2022" which will get installed with the MVSC package we grabbed earlier. This is a preconfigured shell which has all environment variables we need.

Open this up, navigate to the folder with our C source file, then we can create an object file with the following command:

cl /nologo /MD /W3 /O2 /D WIN32 /D WINDOWS /D NDEBUG -I"C:\Path\to\apache\include" /c mod_hello_apache.c

You can read about what these different command line arguments do on the Microsoft documentation. Now we should have a file called mod_hello_apache.obj. We aren't quite done yet. Now we just need to link the Apache libraries to the file:

link kernel32.lib ws2_32.lib "C:\Path\to\apache\lib\libhttpd.lib" /nologo
/subsystem:windows /dll /machine:x64 /out:mod_hello_apache.so mod_hello_apache.obj

You can check out what these linker arguments do too on the Microsoft documentation.

We now have a compiled Apache module called mod_hello_apache.so.

Move it into Apache's modules folder. The last thing we need to do is add this to the bottom of the Apache configuration file in conf/httpd.conf:

LoadModule hello_apache modules/mod_hello_apache.so

<Location "/hello">
SetHandler hello_apache
</Location>

The first line tells Apache we want to load a module called "hello_apache" and provides the path to it. Then inside the location tags we're telling it to use our module for all requests in the URL "/hello".

Now open up your favourite browser (mosaic, of course), and head to the URL "localhost". You should get the same message from before. However, change the url to "localhost/hello" and you should see the words "Hello from apache module!" in beautiful, big, bold times-new-roman.

Well done! You've created your first Apache module! Now you can do your web development in C like it's 1995.

Printing a file with the Apache Portable Runtime

2022-07-31T00:00:00Z

In my last post, I wrote about building an Apache module written in C.

In researching how to do this, I learnt about a the Apache Portable Runtime.

What's that?

It's a number of libraries and header files provided by the Apache Software Foundation which abstract away operating system operations, and allow your C code to become more portable. The Apache Portable Runtime (APR) was originally part of the Apache Server codebase. It got separated out, because having a way to write C code in a single codebase that you know you can compile for multiple platforms is a pretty nice thing to have!

Portability #

C was meant to be a portable language when it was created.

Unfortunately, it was never "write once, run anywhere", as operating systems offer different features and APIs, compilers act completely differently, and C simply grew up in the Wild West days of computing.

Some examples of the issues APR smooths over:

Windows processes are more expensive to create than in Linux.
Unix-based systems and Windows have completely different Socket APIs for TCP connections.
The C language even has undefined behaviour so some functions like fflush(stdin) are completely non-portable.

Pools #

The APR also provides an API to make memory management easier in C.

It uses a system of pools, where every piece of dynamically created memory gets assigned to a pool, which will remember what you've created. Then when you're finished with the pool, you tell it you're done by calling apr_pool_destroy.

After you call this, everything is freed! You don't have to go through all the pieces of data you've dynamically allocated to make sure that they've all been freed and the space has been given back to the operating system. The pool does that for you.

Getting set up #

Although I've just been raving about the portability of APR, these instructions are going to be based on compiling on and for 64 bit Windows, so if you're on Mac or Linux, some of these steps won't apply for you.

So how do we write a program which using the APR? First we're going need a compiler. If you're on Windows I've got some instructions on where to go to get the command line Visual Studio Compiler for C.

Next we're going to need the APR library and header files.

If you have Apache Server installed - you should already have them. You'll find the header files in the include folder and the library files in the lib folder. Both live in the root folder of apache.

If you don't already have Apache downloaded, you can either download the APR source and compile it (if you want to go through that). Or you can grab Apache for Windows from Apache Lounge.

Now open your favourite text editor. Mine is VSCode.

Boiler plate #

First we're going to need to include the APR libraries, and also we're going to grab the classic standard IO library:

#include <apr.h>
#include <apr_pools.h>
#include <apr_file_io.h>
#include <stdio.h>

These will pull in all the APR functions from these header files.

Now we'll write our main function:

int main(){

}

We start by initialising APR and creating our pool.

int main(){
  apr_initialize();
  apr_pool_t *pool;
  apr_status_t status = apr_pool_create(&pool, NULL);
  if(status != APR_SUCCESS){
        printf("Could not create pool");
  }
}

You'll find that usually C frameworks won't be set up for you; you'll have to call a function to start things up. There's no free lunch with a low level programming language. Here we need to call apr_initialize before we're allowed to do anything with APR functions.

Next we declare a pointer to a pool, then pass it to apr_pool_create. This initializes the pool for us. The second argument of apr_pool_create is the parent pool (APR gives you the ability to hold a hierarchy of pools where a parent pool can clean up its children). In our case, this is our first and only pool, so we pass NULL as the second argument.

Finally, it's always good practice to check if things have gone wrong. C doesn't have nice ways to do this like more popular programming languages today. Instead, APR functions will return an enum to represent their status. If things went well, we should see APR_SUCCESS.

Opening a file #

Now let's set up a file descriptor. This is an APR structure which will be able to use to read a file.

  apr_file_t *file;
  char *fname = "C:/path/to/a/random/file.txt";
  status = apr_file_open(&file, fname, APR_READ, APR_OS_DEFAULT, pool);
    if (status != APR_SUCCESS)
    {
        printf("Could not open file");
    }

First we declare a pointer to the file descriptor, then declare our file path as a pointer to the first char in the path, calling it fname. We use the & "Address of Operator" when passing the file descriptor to apr_file_open. apr_file_open's first argument isn't a pointer to a file descriptor. Instead it's a pointer to a pointer to a file descriptor. I don't know why it isn't just a pointer to the file descriptor, but there you go. The second argument is our file name. The third argument is the mode we want to open the file with.

Setting up a buffer #

Now that we've connected with our file, we still need to do some set up before we can read it into a string.

We need to allocate the required memory for where our string is going to be held. This is because we don't know the size of the file and so have to do this dynamically. So our first step is to find out how big the file is, then we can set up the space in memory (a memory buffer) to store the string of the file's contents.

Here's how we do this:

  apr_off_t offset = 0 ;

  apr_file_seek(file, APR_END, &offset);

  apr_size_t size = offset;

  char *buffer = apr_pcalloc(pool, size +1);

  offset = 0;

  apr_file_seek(file, APR_SET, &offset);

What's going on here? apr_file_seek is a function with various uses, but its main use is to move the cursor to a certain place in a file (the cursor is a number representing the current char we're reading from the file).

We want to find out how large the file is, so we move the cursor to the end of the file with the first call to apr_file_seek. The third argument is a pointer to an offset, which will change the cursor to a certain location by its value, but can also be used (as in this case) to get the location of the cursor after the function call. The function sets it to the current location of the cursor after it has changed position. Lastly, the second argument lets us tell it to move to the end of the file using the enum APR_END. After this offset will hold the length of the file in bytes (as a single char is a byte).

Now we create a size variable of type apr_size_t which is a long number that will represent the size of the file. We allocate our memory to hold our string in by setting a pointer to a char variable to the return value of a new APR function: apr_pcalloc.

This function allocates all the memory we need, and sets it all to zero. This is good because C strings need to end with a null character (which is the same value as zero), so the language knows when the string ends. This is also the reason we add one to the size, as we need that extra null character at the end. The first argument is which pool we want the memory to be under.

If we read the file now, the cursor would still be right at the end and we wouldn't get anything back from reading it. For this reason, we need to set our offset to zero, then use the same function from earlier, but this time use the APR_SET enum, which sets the cursor to the offset we provide (which is zero). I personally would probably prefer a different function that would just set the cursor to zero, but hey, it's C. What are you gonna do?

Reading the file #

Now it's time to read our string into the buffer, and print it to standard out!

  status = apr_file_read(file, buffer, &size);
  if (status != APR_SUCCESS)
  {
      printf("Could not read file");
  }
  else
  {
      printf(buffer);
  }

It's as easy as that! We pass apr_file_read the pointer to our file, the pointer to our buffer, and the pointer to our size (this one we hadn't previously stored as a pointer so need to dereference it with &). APR will copy the contents of the file into the buffer, then we're ready to print it out.

Cleaning up #

After we're finished, we need to close our file and terminate APR:

 apr_file_close(file);
 apr_terminate();
 return 0;

The entire source #

Here's what the source should look like, with all the checks and balances:

#include <apr_general.h>
#include <apr_pools.h>
#include <apr_file_io.h>
#include <stdio.h>

int main(){
  apr_initialize();
  apr_pool_t *pool;
  apr_status_t status = apr_pool_create(&pool, NULL);

  if(status != APR_SUCCESS){
        printf("Could not create pool");
  } 
  else { 
    apr_file_t *file;

    char *fname = "C:/path/to/a/random/file.txt";;

    status = apr_file_open(&file, fname, APR_READ, APR_OS_DEFAULT, pool);
    if (status != APR_SUCCESS)
    {
        printf("Could not open file");
    }
    else {
      apr_off_t offset = 0 ;
	
      apr_file_seek(file, APR_END, &offset);
    
      apr_size_t size = offset;
      char *buffer = apr_pcalloc(pool, size +1);
    
      offset = 0;
    
      apr_file_seek(file, APR_SET, &offset);

      status = apr_file_read(file, buffer, &size);
      if (status != APR_SUCCESS)
      {
          printf("Could not read file");
      }
      else
      {
          printf(buffer);
      }
      apr_file_close(file);
    } 
   }
   apr_terminate();
   return 0;
}

Compiling the program #

Now save the file as aprprint.c.

Then you can compile the program on x64 Native Tools Command Prompt by navigating to your source directory then typing the following command (replacing the paths with the paths to your Apache folder):

cl /nologo /MD /W3 /O2 /D WIN32 /D WINDOWS /D NDEBUG -I"C:\path\to\apache\include" aprprint.c kernel32.lib ws2_32.lib "C:\path\to\apache\lib\libaprutil-1.lib" "C:\path\to\apache\lib\libapr-1.lib" /nologo

Now you should get back silence from the compiler. Have a look at the error messages if your don't.

The last thing to do is to add the bin folder in apache to your environment variables path. This will mean the DLL's that Apache provides will be in scope for your program.

After all this, you can run the program with:

aprprint.exe

You're an APR programmer now! 😈

More! #

Try adding the file name as a command line argument to print off different functions. Or you could combine this program with my last post about Apache modules to make a static file server.

Here's a tutorial from Inoue Seiichiro about the APR which goes more in depth than I can.

Hosting small websites on EC2

2022-10-02T00:00:00Z

You have a small site where you don’t expect many visitors, but you want dynamic server rendering and you want it cheap. My new favourite hosting for this use-case is Amazon Web Services’ EC2.

You might struggle with the fact you're giving the richest people in an unequal world more money. And maybe your time is worth too much to waste on server configuration. But if you have a small passion project, a cheap EC2 instance provides the most freedom for the lowest price.

AWS don’t heavily market how to do things cheaply. At the end of the day, it’s in their interest to push higher margin platform-as-a-service over dumb infrastructure.

AWS won’t give you detailed help with setting up your stack. There's too much to document for a rented computer which can do anything you want it to. Although, they make a valiant effort to document the most heavily trodden cow paths.

The secret is that you can install all your stuff on EC2. And on the cheapest type of instance. You can save a lot of money and complexity. Basic building blocks like EC2 help us bring back mildly dynamic websites.

It doesn’t even seem like it’s in AWS’s interest to bother to cater to cheapskates like me. And yet they serve my use-case better than any other hosting provider. I guess 15 years of running infra-as-a-service means they’re able to automate the sales, admin and maintenance away. And they benefit from an economy of scale of biblical proportions.

What type of website are we talking about? #

What is the use-case I’m talking about? If you want the absolute cheapest possible website, you’re going to want a static site. There’s a smorgasbord of cheap and free places to host static HTML, CSS, and JavaScript (github pages and neocities come to mind). The site you’re reading from now is all static files in azure storage, and I wrote a blog post about moving here.

But only running static files is limiting. I want to make websites that do things. You might yell at the screen: “use JavaScript” and “use a service”. But we can’t write everything in JavaScript: There’s always going to be a point where we have privileged data, or when we risk a spinnaggedon request waterfall. I’ve enjoyed making a lot of apps that only run on the client. But there are a lot of apps where this just doesn’t make any sense — see Jason Miller’s Application Holotypes.

And services? We’re trying to be cheap here! If I can pay $5.00 a month for a server, why would I want to pay another $5.00 a month for comments, another $5.00 a month for authentication, and another $5.00 a month for analytics. There’s a point where this is worth it, but it isn’t your side project. EC2 gives you a virtual machine that you can use for whatever you want.

Spinning up an instance #

I wanted to spin up a quick PHP website which didn’t need a database - just a JSON file for data. I created a T2.nano instance, using a gp3 volume for storage (this is like the hard drive for the VM). I went with Amazon Linux 2 for an operating system - I assume this OS has been optimized for the cloud. The marketing certainly says that it has been.

Now to make our instance do something, we need to connect to it. There are ways to connect to the machine graphically, but we’ll use SSL because I like to cosplay as someone good at computers.

After you create an instance, you’ll be prompted to download a .pem key. Keep this safe. If you lose it you won’t ever be able to connect to your EC2 instance. By default, the username on Amazon Linux will be ec2-user. And you need the Public IPv4 DNS. You can find it in the instance details.

Then, to connect to a fresh Amazon EC2 instance via SSL from windows PowerShell:

    ssh -i C:\path\to\mykey.pem ec2-user@my-instance-public-IPv4-address.amazonaws.com

You might be prompted for some trust warning or something which I said yes to without understanding what I was agreeing to.

Now you should get some ASCII art courtesy of Adam Selipsky.

Installing stuff #

You’re in. Let’s get Apache:

    sudo yum install -y httpd

This will use a Linux package manager called yum to download and install Apache. After this has finished, you can start Apache with:

    sudo systemctl start httpd

And test it with:

    curl http://localhost:80

And stop it with:

    sudo systemctl stop httpd

Great. Now you can grab and install PHP.

If you’re like me, you probably want nice URLs. You can do this with Apache’s mod_rewrite module, and redirect all traffic to index.php by editing /etc/httpd/conf/httpd.conf. Edit it over SSL by using a command-line text editor like nano. Add the following to the end of the config file:

    #This did not work for me
    <IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond %{REQUEST_FILENAME} !-d
        RewriteCond %{REQUEST_FILENAME} !-f
        RewriteRule . index.php [L]
    </IfModule>

Wait that didn’t work for me (even though it worked for everyone else on stack overflow). For me, (and maybe you), this was what I needed:

    #Redirecting all input to index.php
    <IfModule mod_rewrite.c>
        RewriteEngine on
        RewriteCond /path/to/document/root/%{REQUEST_FILENAME} !-f
        RewriteCond /path/to/document/root/%{REQUEST_FILENAME} !-d
        RewriteRule ^(.*)$ /index.php [QSA,L]
     </IfModule>

Now you can make your own little PHP router like this one from John O. Paul. And you’ll need some way to get your code onto the machine (I’ve just been pulling down from github using git but I’m sure there’s a better way . . . github action?).

You might run into another problem. Apache won’t have the rights to write files! There are some instructions on the setting-up-LAMP page on the AWS site for how to sort this out. It involves a command called chown which might be my favourite name for a command.

And we all love locks to the left of our URL bar. We better buy a domain, set up Amazon name servers, and also get a TLS certificate from let’s encrypt and install it into Apache.

Before you can connect with a site hosted on your EC2 VM, you’ll also need enable traffic from all IP addresses using Inbound rules.

The counterargument #

Wait. Those were a lot of steps. And for a site that might not scale very well? Was this all worth the time and effort? Was the cheapness really worth it? How much maintenance is this all going to require? What about security? Is everything configured properly in a way where no one’s going to steal my data?

These are all extremely valid questions. And they are the reason you might just want to run away and host a site on Netlify, or try AWS amplify if you’re still desperate to give Bezos more bucks.

However, the nerdy side of me enjoyed getting this all working, and (I hope) I’d be faster at setting it all up the second and third times I do this. I really like the idea that I can have a deployment environment which isn’t limited by what the powers-that-be decide I’m allowed.

And I love that it’s cheap . . . 🤷‍♂️

Gizmo Girls

2022-12-01T00:00:00Z

I've started a new blog! Gizmo Girls baby!

I wanted to Spend all my innovation tokens and really try out a whole bunch of cool new tech. I also wanted a set up where someone non-technical would be able to make changes. Another thing I wanted was to be able to update the blog from my phone. This meant that the workflow for this blog you're reading (writing markdown files and pushing them up to github, then letting a github action deploy to azure blob storage) wouldn't work.

I also really wanted to try out astro.js. I love that it's component driven, HTML-first, and it's super fast. Astro templates are what JSX always should have been. And the model of only opting into client-side javascript is one that is going to save on load times and complexity.

Like all my side projects, I also didn't want to invest a whole bunch of cash in something that I don't expect to make any cash FROM. Here's what these requirements looked like:

Requirements #

Astro.js
Able to be edited and updated by someone non-technical
Able to be edited and updated from a mobile device (eg. one that doesn't need node.js installed)
Cheap $$$

Airtable #

As I planned to run this all as a static site, I knew immediately that I wouldn't be updating the content from within the site - as you might do with non-headless (head-full?) WordPress. So I knew that I'd have to go to an outside service. So I went with airtable.

Airtable is sort of like google sheets, but you can much more clearly define what data should go in each cell, and also have options for allowing rich text and images.

They've got great API documentation, where code snippets will be dynamically generated based on your data - absolutely killer. And the API is allowed in the free tier. I can't recommend them enough. They also provide a JavaScript package you can use to hit their API to make things super smooth.

Github actions #

With the blog you're reading now, the only way I can change the site is to push up some code to the main branch on github. This will fire up the github action and the static files will get pushed up to their new home.

However it is also possible to manually run a github action. I did this for Gizmo girls. What the github action does is downloads all the data and images from airtable, runs the astro build process, then pushes it up to an Azure blob storage static website. This way if I need to edit the site, I can go to airtable, make my edits, then go to github and press "build". All from the comfort of my phone. Nice.

Todo #

There are a few limitions and things that could be better:

Images - At the moment there's only one image per post on there. Airtable's rich text doesn't allow you to inline images so I might need to create anothe table to hold the images and my own custom sytax to convert them
Caching - The blog will download ALL posts and ALL images on every build. That isn't gonna scale. Eventually I might add some logic in to only grab what has changed.
RSS - The RSS feed on the site is just links to posts at the moment. I want to figure out how this could be converted to a feed which includes the full post.

Goodbye Heroku free plan

2023-01-08T00:00:00Z

A few of my posts on here have been about coding on the cheap. Static sites, EC2 instances with everything installed on them, Github actions, free plans. Sometimes it’s to my detriment as I sacrifice a whole lot of my time for a few less dollars per month. Buuuuuut my side-projects haven’t been aimed at making money, so I don’t want them to suck up money. Maybe after a few more years on a developer salary my priorities will change and I’ll shoot for options that save time over cash. But I also enjoy the limits I set myself. They narrow down the infinite options we have for building and hosting sites.

These last few months have been a sad time for the web app cheapskates out there. We’re all slaves to capitalism — especially companies with “sales” in their name. And on 25 August 2022 Heroku (now a subsidiary of Salesforce) announced that they would be ending their free plan. In their announcement they advised that starting 28 November 2022 they would start shutting down free-plan dynos. The app I wrote to learn Laravel, Plantr, was hosted on a free Heroku dyno. I started revisiting the site throughout the end of 2022. It continued serving content into December, but a couple of weeks ago I hit the site and got an error message from Heroku.

Building that app over the last quarter of 2020 introduced me to a whole bunch of concepts; I learnt about ORMs, Alpine.JS (even got a CSS tricks article showing off my form validation), dependency injection with controllers, and Composer. Using a Rails-style model-view-controller framework back in 2020 has come in quite useful now that I have a job at a C#/ASP.NET shop (I haven’t blogged about the .NET ecosystem, however I might start out of pure frustration).

I didn’t have a whole lot of proof that I knew how to code before I uploaded Plantr - maybe a couple of JavaScript games and this website. Now I’ve had a job in web development for almost a year, and I’ve written a few CSS Tricks Articles there’s slightly less pressure for me to run side projects to prove I can code. I will still do side projects but they’ll be for fun now.

I understand that expecting for-profit companies to continue hosting server-rendering applications is a bit of a tall order — even in this world where the cost of compute increasingly approaches zero. In Adam Wiggins’ interview on the ChangeLog he talked about how there were a whole bunch of hobby apps on Heroku and it was difficult to tell which free apps were actually important to people. Servers need to grind away to intermittently fire them up, and it takes storage to keep them. If there’s a whole bunch of one-off apps ready to run at any moment, that costs Heroku cash, possibly with zero benefit.

At the same time, it’s a little sad that my side-project is no longer out in the wild to show off, even if it might be to laugh about how much (or how little) I’ve learnt. Maybe one day I’ll give it a spruce-up and a few more features and upload it somewhere else again. If so, it would be my third attempt at making it. I don’t know if that’d be the best use of my time, so for now it's going off to application heaven. Plantr served its purpose for me as a self-teaching tool. I don’t think I want to pay $60.00 a year to keep it running somewhere else.

The web is such a democratic platform and I hope that the death of free plans doesn’t lock too many people out. Coming from New Zealand I’m far more economically privileged than a lot of the world. It’s hackers poorer than me we should worry about. But I reckon it's going to be okay: whichever platform provides the free plans and creates the least friction is the one that will going to get adoption.

That's the platform that’s going to be sustainable.

Hill Times, Vim and Delayed Gratificication

2023-01-31T00:00:00Z

There's a hill near my house, about ten minutes walk away. A steep windy road goes up for five more minutes of (hot) walking, then a gate takes you out of suburbia and into the bush. From there on it's a steep firebreak. The soil is a light brown clay.

It gets slippery on rainy days, and it's broken and uneven. Going up the hill you have to duck through mānuka in some sections or tip-toe around mud. Eventually you make it to the top and you can look out over Lower Hutt, Wainutomata, Wellington harbour.

On a clear day you can look out into the distance and even see the Kaikoura ranges to the South. The hill is about 250m high according to the topological map.

I ran up that hill 48 times in 2022. I timed myself each time. Here are the times on the different days:

Time (minutes)	Date
21.08	14/1/22
19.42	17/1/22
19:26	19/1/22
19:51	26/1/22
19:38	3/2/22
19:18	15/2/22
20:00	3/3/22
19:05	7/3/22
18:41	22/3/22
18:08	28/3/22
19.31	10/4/22
18:20	13/4/22
19.29	25/4/22
19:54	28/4/22
18:39	1/5/22
18:17	3/5/22
19:49	22/5/22
19:24	23/5/22
19:06	28/5/22
18:10	29/5/22
20:46	8/6/22
20:52	3/7/22
23:01	28/7/22
20:43	31/7/22
21:45	7/8/22
21:00	22/8/22
21:07	23/8/22
20:17	25/8/22
20:28	28/8/22
21:42	30/8/22
20:10	31/8/22
20:23	2/9/22
21:53	19/9/22
19:20	21/9/22
20:21	25/9/22
21:00	28/9/22
19:46	2/10/22
18:38	3/10/22
18:34	10/10/22
26:46	15/10/22
17:57	16/10/22
19:09	22/10/22
18:58	30/10/22
18:02	1/11/22
17:38	8/11/22
22:11	11/12/22
21:25	12/12/22
19:57	15/12/22
19:46	23/12/22

This is the way I get myself to do excercise. The timing means I have something to work towards - making that number go down. I also love get to see the views, and how things change as the weather and seasons change. And so many different times of the day (I'm very inconsistent with what time I run). It's also good time for listening to podcasts - although I probably don't pick up much information, wheezing away as I try to ascend up the clay.

The main reason I do this is for my mental health. It helps a lot with that.

Unfortunately, I can't lie and tell you there isn't a part of me that says I'm running up that hill to look good. But I avoid thinking about it like that. I don't want to hate my body or have a negative relationship with excercise or food.

Regardless, I'm proud of making it up the hill that many times in 2022. I'm looking at my calendar now and I haven't made it up once this year and January's almost already over. It might be time to get started again.

Running uphill is agony, but afterwards I only feel the endorphins. And when I'm 68 I'll feel that much better from the excercise I'm doing today. It reminds me of another piece of delayed gratification I'm dealing with:

Vim #

Vim is a text editor. But not a normal one by any means. Notepad is to vim as a spade is to the Kola Superdeep Borehole drill. It's a difficult tool to drive, but it holds a seismic power. Yes there's jokes about quitting (:qw to quit and save, :q! to quit and throwaway your work). Yes there's a stereotype that this is the tool of the neckbeards and sysadmins.

But I really want to look after my hands as I code. Someone who's good at vim has the most terse way of getting letters into the computer. They most certainly don't need to bother with their mouse like us peasants. Letter keys are used for different commands and naturally live where your hands fall on the keyboard. Using short commands to update repetitive changes makes a lot of sense.

You can hear a great podcast on why people like vim on the changelog. A good place to start learning vim is with vimtutor. It's where I started and takes you through the basics of how vim works. Then I tried to start using vim consistently and googling things that I needed to learn.

The problem with the gratification of vim is that like running up a hill, the gratification is delayed. Some things chafe, especially when you're looking for your file explorer to the left of your screen, and when you're used to windows and you control+v to paste (WRONG!). And the vimscript syntax has so many hidden secrets. I still seem to need stack overflow for every one!

Finally (and I know this is my fault), by default vim is ugly. It has stark brutalist colours and terminal fonts. Listening to Caleb Porzio on syntax FM talking about how obsessed the laravel community are on code beauty and developer happiness reminds me that we deserve for things to look nice.

I guess the main thing is that I could configure all this and learn all that but it takes time and effort. At the moment it's a case of me still liking VSCode because it's what I know.

DotNetCore, Azure AD, and SAML

2023-04-20T00:00:00Z

DotNetCore, Azure AD, and SAML are an odd mix of old and new tech I found myself needing to combine. You’ve got the newer cloudiness of Azure, the fresh C# runtime DotNetCore, but a crusty old authentication protocol with SAML.

If you’re like me and have been working on a (mostly) greenfield C# project and find that you need to authenticate to Azure AD using SAML, this guide might come in useful for you. There is nothing on the internet about this specific mix of application runtime, identity provider and protocol. Although there is this guide from Okta on using DotNetCore with their own SAML authentication, which I am partly basing this article off.

First, a little explanation of these three technologies:

DotNetCore (also spelled .NET Core). This is a runtime Microsoft provide, usually for C#. Like all things Microsoft provide, it’s confusingly named. This is a different product from .NET Framework. We’re specifically going to be looking at a web application built with DotNetCore. If you need advice on using SAML authentication with .NET Framework, you’re going to find a lot more articles on this (because they’re both old).
Azure AD (Azure Active Directory). This is Microsoft’s way of giving you the ability to know who someone is when they hit your application. It gets used by schools and businesses to keep a list of people who should be able to log into internal apps. It means we don’t have to keep any passwords or usernames. Microsoft is going to do all that dirty work for us.
SAML (Security Assertion Markup Language) is not a Microsoft product. This is an open standard for authentication. It was created in the early 2000s when XML was very cool. Most of us have fortunately moved on from XML and now use JSON because we want to enjoy our lives. Unfortunately, the enterprise ecosystem hasn’t. There is a lot of jargon that gets used to SAML protocol. I’m going to avoid using it as much as possible in this article because it makes my eyes glaze over. Basically it’s a way for you to be able to redirect the user to a 3rd party who holds their credentials, then allow that 3rd party to come back and tell us who the user is. In this case, the 3rd party is Microsoft.

Scaffolding A DotNetCore Web App #

You can skip this part if you have an existing DotNetCore app you’re trying to get authenticated, although it’s probably good to take a look at the architecture I’ll be using for the example.

To keep things simple, we’re going to use a server-rendered razor pages app (although it is easily possible to use these technologies in a single page application with React or Angular). You’ll need to have DotNetCore installed, and the dotnet CLI added to your path. To scaffold a server-rendered app with DotNetCore, navigate to a new directory and use this command:

    dotnet new razor

Then you should be able to immediately start your app with:

    # Generates a developer https certificate
    dotnet dev-certs https
    # Starts app
    dotnet run

You should be able to hit the localhost URL it spits out (mine defaults to https://localhost:7269) and see a welcome page.

Setting Up Azure AD #

You can skip this part if your company has an infrastructure department who are going to do this for you.

In this part, we’re setting up the Microsoft Active Directory service which will know that we are going to be writing an application and we want to authenticate with SAML.

Step one: get an Azure account if you don’t have one.
Step two: head to https://portal.azure.com and searching “Azure AD” in the search bar and click into “Azure Active Directory”.
Step three: click into “Enterprise Applications” on the left. Now search for “azure ad SAML toolkit” and click this type of app to create your new application registration.
Step four: Click into your new application, then into “Single Sign On” on the left. Finally select “SAML”.

Now you need to click “Edit” on the first block of items - the “Basic SAML Configuration”. Here we need to enter three things:

Identifier - This is an arbitrary string which identifies which of your applications you are trying to authenticate (in case you have more than 1 application). It can be literally anything, but it’s pretty normal to make it the URL of your site. I’m going to make it “myidentifier”
Reply URL - this is where Microsoft is going to send the user back to after they’ve authenticated by using a POST redirect when the user signs in with Microsoft. More on this later. I’m going to use the URL: https://localhost:7269/Auth/SamlResponse. You would usually give it a few — one for dev and one for production
Sign On URL - this is the URL for our application - so that Microsoft knows that the request is coming from the right place. I’m going to use the URL: https://localhost:7269.

Getting Your Certificate #

Now you can scroll down and download the “Certificate (Raw)”. This certificate is something our app needs to tell that the user has been authenticated by Microsoft. It’s important to keep this safe, as if somebody steals it, they can pretend that they have been logged in by Microsoft when they haven’t been.

Save the certificate in the root folder of your application as cert.cer and add it to .gitignore.

Getting Your Metadata URL #

There’ll be another item on the SAML toolkit page where the name is “App Federation Metadata Url”. Copy this URL and keep it somewhere. Basically what this is is a link to an XML description of the SAML API Microsoft is providing, with a whole bunch of data that our app needs to understand and verify what we get back from Microsoft.

Adding Yourself As A User #

Now you’ll need to add yourself to say you can get into the app. From the Azure AD SAML Toolkit enterprise application, click “Users and Groups” on the left. Then click “Add user/group”, then click “None selected” under users. Now you should be able to add yourself to the app.

ITfoxtec SAML #

We’re going to go ahead and use a nuget package from the folks over at ITfoxtec: ITfoxtec - ITfoxtec Identity SAML 2.0. This is because implementing your own SAML client would be a horrible, time consuming experience. You can install the package with these commands:

    dotnet add package ITfoxtec.Identity.Saml2
    dotnet add package ITfoxtec.Identity.Saml2.MvcCore

Now we can actually start writing some code. Edit your program.cs file. It should be on the root level of your application folder. First add the imports at the top:

    using ITfoxtec.Identity.Saml2;
    using ITfoxtec.Identity.Saml2.Schemas.Metadata;
    using ITfoxtec.Identity.Saml2.MvcCore.Configuration;

Then lower down, just before it says app.UseAuthorization(); add in:

    app.UseAuthentication();

Then scroll back up to builder.Services.AddRazorPages(); and add the following config to register that you want to use ITfoxtec for auth:

    builder.Services.AddRazorPages();
    ConfigurationManager configuration = builder.Configuration;
    builder.Services.Configure<Saml2Configuration>(configuration.GetSection("Saml2"));
    builder.Services.Configure<Saml2Configuration>(saml2Configuration =>
    {
        saml2Configuration.AllowedAudienceUris.Add(saml2Configuration.Issuer);
        string rootDirectory = configuration.GetValue<string>(WebHostDefaults.ContentRootKey);
        var cert = new System.Security.Cryptography.X509Certificates.X509Certificate2(rootDirectory + "\\cert.cer");
        saml2Configuration.SignatureValidationCertificates.Add(cert);
        var entityDescriptor = new EntityDescriptor();
        entityDescriptor.ReadIdPSsoDescriptorFromUrl(new Uri(configuration["Saml2:IdPMetadata"]));
        if (entityDescriptor.IdPSsoDescriptor != null)
        {
            saml2Configuration.SingleSignOnDestination = entityDescriptor.IdPSsoDescriptor.SingleSignOnServices.First().Location;
        }
        else
        {
            throw new Exception("IdPSsoDescriptor not loaded from metadata.");
        }
    });
    builder.Services.AddSaml2();
    var app = builder.Build();

Appsettings.json Configuration #

What the code above is doing is grabbing a bunch of stuff from appsettings.json and giving it to the ITFoxtec library. We need to fill appsettings.json with data from Azure AD so that ITFoxtec knows how to connect to Microsoft’s SAML service. Create a new key in the top level JSON called “saml2” pointing to a JSON object with 3 keys: “IdPMetadata”, "Issuer", and "CertificateValidationMode”. In “IdPMetadata”, add the metadata URL we grabbed earlier. In “Issuer”, add the Identifier from earlier (yes it bothers me these have different names). Finally in “CertificateValidationMode” add “None”.

This is what mine looks like (with the actual GUIDs replaced with “a-random-GUID”):

    {
      "Saml2": {
        "IdPMetadata": "https://login.microsoftonline.com/a-random-GUID/federationmetadata/2007-06/federationmetadata.xml?appid=a-random-GUID",
        "Issuer": "myidentifier",
        "CertificateValidationMode": "None"
      }
    }

Why are we putting CertificateValidationMode as none? That sounds a bit gung-ho with security? I can assure you that it’s not. Basically, what is going on here is that there’s a cartel of companies and organisations who create certificates for SSL and have programmed a large amount of software (including your browser) to get angry if certificates aren’t made by them. CertificateValidationMode is a way to tell ITFoxTec to get mad at you for using a non-supported certificate. The thing is, that the Azure AD certs aren’t part of the certificate club, so we need to set this to false to not throw an error.

Adding Controller Routing #

Now we’re going to need a controller. This means we need to change some of the routing from the default for server-rendered razor apps. Replace this part of the config:

    app.MapRazorPages();

With this:

    app.UseEndpoints(endpoints =>
    {
        endpoints.MapRazorPages();
        endpoints.MapControllerRoute(
            name: "default",
            pattern: "{controller=Home}/{action=Index}/{id?}");
    });

This is going to tell it to try and find a razor page to match the route, then if it can’t find it, it will look for a controller using the format controller/action/id where ID will be passed to the action method if it asks for it. This is a pretty standard way to do controller routing these days.

Now our app should still be work and won’t error out when we hit the home page.

Writing An Auth Controller #

Create a new file called AuthController.cs and fill it with this:

    using ITfoxtec.Identity.Saml2;
    using ITfoxtec.Identity.Saml2.Schemas;
    using ITfoxtec.Identity.Saml2.MvcCore;
    using System.Security.Authentication;
    using Microsoft.AspNetCore.Authorization;
    using Microsoft.AspNetCore.Mvc;
    using Microsoft.Extensions.Options;
    namespace AzureAd
    {
        [AllowAnonymous]
        [Route("Auth")]
        public class AuthController : Controller
        {
            const string relayStateReturnUrl = "ReturnUrl";
            private readonly Saml2Configuration config;
            public AuthController(IOptions<Saml2Configuration> configAccessor)
            {
                config = configAccessor.Value;
            }
            [Route("Login")]
            public IActionResult Login()
            {
                var binding = new Saml2RedirectBinding();
                return binding.Bind(new Saml2AuthnRequest(config)).ToActionResult();
            }
            [Route("SamlResponse")]
            public async Task<IActionResult> SamlResponse()
            {
                var binding = new Saml2PostBinding();
                var saml2AuthnResponse = new Saml2AuthnResponse(config);
                binding.ReadSamlResponse(Request.ToGenericHttpRequest(), saml2AuthnResponse);
                if (saml2AuthnResponse.Status != Saml2StatusCodes.Success)
                {
                    throw new AuthenticationException($"SAML Response status: {saml2AuthnResponse.Status}");
                }
                binding.Unbind(Request.ToGenericHttpRequest(), saml2AuthnResponse);
                await saml2AuthnResponse.CreateSession(HttpContext);
               return Redirect("https://localhost:7269");
            }
        }
    }

There’s a lot going on here. There are two endpoints here; the login endpoint, and the SamlResponse endpoint.

The login endpoint is where the unauthenticated user sends a request to when they want to log in:

            [Route("Login")]
            public IActionResult Login()
            {
                var binding = new Saml2RedirectBinding();
                return binding.Bind(new Saml2AuthnRequest(config)).ToActionResult();
            }

What will happen here is that the SAML library will redirect the user away from our app and over to Microsoft. As it does this, it adds a large amount of data in URL parameter “saml2” which will contain a bunch of information Microsoft asks for in its metadata (including the identifier). It deflates it (applying a kind of compression), base64 encodes it, then URL encodes, then sends it. You can see what the contents contains using this handy tool.

Thankfully this is all abstracted away from us with the Saml2RedirectBinding class, and we just give it the config it needs to go ahead. Here we can also tell Microsoft to loop back to a different domain (eg. between development and production) by setting the AssertionConsumerServiceUrlproperty on the Saml2AuthnRequest.

The SamlResponse Endpoint #

The SamlResponse endpoint is where the user is going to get redirected back to after Microsoft has said they’ve entered in the correct username and password and they’re logged in:\

           [Route("SamlResponse")]
            public async Task<IActionResult> SamlResponse()
            {
                var binding = new Saml2PostBinding();
                var saml2AuthnResponse = new Saml2AuthnResponse(config);
                binding.ReadSamlResponse(Request.ToGenericHttpRequest(), saml2AuthnResponse);
                if (saml2AuthnResponse.Status != Saml2StatusCodes.Success)
                {
                    throw new AuthenticationException($"SAML Response status: {saml2AuthnResponse.Status}");
                }
                binding.Unbind(Request.ToGenericHttpRequest(), saml2AuthnResponse);
                await saml2AuthnResponse.CreateSession(HttpContext);
                return Redirect("https://localhost:7269");
            }

binding.ReadSamlResponse will get all the information from the SAML data, and if there’s no error here we should fully have all the data we need about the user. If you printed the raw POST body as a string then check it with the SAML defalting tool you can see what Microsoft is sending back.

However, even with all this data it’s important to run the unbind method. This checks whether hashing the response with our certificate will get the same result as what Microsoft adds to the “certificate” field in the SAML response. It’s a way to make sure that the information in the response is actually coming from Azure AD. Then if all has gone well we can create the user session, and from here we’ll know who is logged into the app.

User Claims #

The way information is stored about the logged in user is in the User class, which is made accessible to all pages and controllers in DotNetCore apps. What we get is a list of “claims” which are basically key-value pairs of information about the user, which might be their name, email address, or whether they’re a super user. Azure AD will send a bunch of default ones which we can use. You can grab the user claims like this:

    var nameClaim = User.Claims.FirstOrDefault(c=>c.Type=="http://schemas.microsoft.com/identity/claims/displayname");

Let's use this claim to display the user’s name in our app:

Updating The Homepage #

Edit the Pages/Index.cshtml page so the contents of it looks like this:

    @page
    @model IndexModel
    @{
        ViewData["Title"] = "Home page";
    }
    @if (User.Identity.IsAuthenticated)
    {
        var nameClaim = User.Claims.FirstOrDefault(c=>c.Type=="http://schemas.microsoft.com/identity/claims/displayname");
        var name = nameClaim == null ? null : nameClaim.Value;
        <p>Hello, @name</p>
    }
    else
    {
        <a asp-controller="Auth" asp-action="Login">Login</a>
     }

Now we have a functional page which will only display a login link for an unauthenticated user, and will display the users name after they’ve logged in.

Now you should be able to start your application with dotnet run then head to the home page, then click the “login” link.

Moving on from here #

There’s a bunch of stuff that I haven’t done which would be pretty normal in an app:

Creating some middleware that will redirect the user to the home page if they aren’t logged in
Adding some custom claims to the user to give them different access rights to different parts of the application
Make it configurable from appsettings.json which URL out from dev and prod you want Microsoft to loop back to using using the AssertionConsumerServiceUrl property of the Saml2AuthRequest

Sticker Charts

2023-08-13T00:00:00Z

Reporting #

For a large amount of my life I didn't really measure much. I'm bad at arts and crafts because I don't properly cut at the right angle. When I volunteered on farms I'd plant in weird lines. I like to cook a lot more than I like to bake. I’ve always considered myself an “artistic” person more than a “science” person.

Now measurement, numbers and reporting are literally my job at a data-obsessed company. I've been building reports for HR data around recruitment, headcount and attrition, and training data.

It turns out people find data really useful. And by people I mean business people. You need data to see what's working, where things are going, how people are doing at their job. You need data to make the number go up - usually the number is "profits".

Power BI #

There are a number of reporting tools out there. There's Apache Superset, there's Tableau, but the most ubiquitous is Power BI. Power BI is a reporting offering from a certain little company in Seattle. It's is available with a product they sell called Microsoft 365; a SAAS product most businesses already have a subscription with. So it's a no brainer for these businesses to use Power BI. They're already paying for it.

Power BI is a pretty classic Microsoft product: absolutely packed with features thrown together in the most haphazard way you can imagine. It's easy to get started, with but once you get off the beaten path you start to feel bounds of the walled garden.

My 3 big Gripes with Power BI #

As a developer coming to Power BI there's a few big things that annoy me:

Bad Code Editing Experience #

When you want to write a DAX measure or do something custom with M Query on the desktop app you'll find yourself in Power BI's text editor. It has some basic autocomplete, and some small amount of syntax highlighting but that's it. There's no Control-F, there's no way to look up definitions of functions, there's no formatter, there's no way to customise the editor theme in any way.

I’ve had to literally copy and paste an M Query snippet to notepad so I can control-F throught to check whether I'm including a column. You know something is wrong when notepad has a feature your text editor doesn't have.

2 Different Proprietary Languages #

There's 2 different languages Power BI uses: M query and DAX. It's important to have a deep understanding of both if you're doing anything nontrivial in Power BI reports. I'll accept Microsoft will be Microsoft and is allowed to create new proprietary languages, knowing full well that my knowledge of JavaScript, C# and SQL becomes useless in this environment. But what really annoys me is they created 2 new proprietary languages for a single product, and both have completely differing syntax.

For instance in DAX, we have the magical operator “IN” to check if something is in a list:

    NewColumn = IF(
        'Table'[OriginalColumn] IN {"String1", "String2", "String3"},
        TRUE(),
        FALSE()
    )

Whereas in M query we have to use a “List.Contains” function:

     NewColumnTable = Table.AddColumn(
        Source, 
        "NewColumn", 
        each List.Contains(
            {"String1", "String2", "String3"}, 
            [OriginalColumn])
        )

In DAX we have the logical operators && and || for “and” and “or”. Whereas in M Query we use the words and and or for the same task. Wouldn’t it be better for the syntax to be consistent when it’s part of the exact same platform?

Lack of Smart Caching Behaviour #

When you're working on a Power BI dataset, you write your query then it will get pulled it down from whatever source you use (whether that be an SQL server, a Power BI dataflow, a web service or any number of other sources). Then you can add steps to process the data - removing and renaming columns, adding types to them, merging with other queries etc.

The problem seems to be that there's no separation between the grabbing of the data and the processing of the data. And this will mean that every time you change your M Query it will need to redownload your whole data source. This can take up to half an hour with the dataset sizes I've been working with. There's no way to opt out of redownloading all the data to say "no, I don't care that this isn't fresh". No, it will be a half an hour iteration to check whether your code works.

Good Things About Power BI #

I could go on about things that annoy me about Power BI, but some of the good things:

When you're following the happy path, Power BI is a super quick way to throw a report together. It comes with a lot of default visuals and there’s also an ecosystem of third party visuals. Microsoft forums have a lot of advice on different problems. Also, sometimes the low-code features can save you a lot of time writing code. There’s also permissioning tools, usage analytics tools, and a bunch of different integrations with other Microsoft products of course.

Sticker Charts #

Working on these reports also make me wonder whether I should start measuring more things in my life, and presenting these measurements in a visual way. Having a visual way to see your progress in something makes that progress feel more real. This is the reason senior leadership teams and business people are so into reports.

It’s like a kid’s sticker chart: a kid does something right and they get a sticker on their chart. They feel the endorphins of seeing what they achieved, and it makes it a lot easier to do the next time because they had a visual symbol that they did well. I feel like I could use reporting to encourage me to get out of my default behaviour (endlessly scrolling down a social media app). And maybe do something better, like learn vim or run up a hill.

Ways of getting data from Excel files into a Database

2024-01-18T00:00:00Z

The lingua franca of the government/enterprise world for communicating data is the excel file. SQL doesn’t have the same level of adoption for non-technical people. This means that if you’re an SQL person (As I'm attempting to be), you’re bound to get asked to translate between these two forms of tabular storage - from excel to an SQL db and vice versa. Often a big part of my job is that I get sent an excel file from a non-technical person, then I need to make bulk updates in a database based on this data.

I find that my preferred method to do bulk updates is to smash the data into a table (you can use a temporary table, but I prefer using a regular table, then delete it after a few days) as opposed to writing a single script and running it as a one-off,. This allows me to make some assertions on what I expect the updates to look like (eg. how many updates I’m about to do) then I can execute when I’m confident about what the result will be.

Before I go on: I should note that the flavour of SQL I’ll be looking at here is that of Microsoft SQL Server. If you work in government or enterprise, this is likely the flavour you’ll be dealing with too.

So without further ado, here are some ways to get data into a database from an excel file:

SSMS flat file import #

This method is quite convenient and requires the least typing as you get to use a graphical user interface to get the data into your database.

The first step is to save the excel sheet you want to import as a CSV file. Just the regular default save settings for saving as a CSV work for me.

Now open up SSIS, then navigate to your database. Right click your database, click into “Tasks” then “Import Flat File…”. Get the path to the import file, configure the table name (it will have to be a new table; this method doesn’t seem to allow appending to tables). Then in “Modify columns” tab you can configure the types for each of the columns.

Click through to “Summary”, then “Finish”, then refresh your database and you should be good to go.

Using the import/export wizard - Microsoft Access #

This is another way to insert from an excel into a database where you don’t need to convert your excel file to a CSV first. It also means you can do multiple excel sheets at once, and with this method you can append to a table instead of only being able to create a new table. There do seem to be issues with this method: I’ve found that you have to download some tools to get this going, and even then, it seems to periodically not work for reasons that aren’t entirely clear to me.

The first step you’ll need to take is to download the Microsoft Access Database Engine Redistributable. Also you need to get the 32 bit version - not the 64 bit version.

After you have Microsoft Access installed, we’re going to do something similar to the flat file import: open up SSIS, right click your database, click into “Tasks” then “Import data…”. If it’s your first time here, you should see the title “Welcome to SQL Server Import and Export wizard”.Click “Next” (or alt N if you’re too cool to use a mouse).

Choosing a data source

In the dropdown, click into “Microsoft Excel”, it should have an ancient Excel logo (one which immediately transported me back to my childhood in the 90s). Then add the path to your excel file and select the excel version you have on your computer. Click “Next” (or alt + N).

Choosing a destination

I choose “.Net Framework Data Provider for SqlServer” for a destination. Unfortunately, this doesn’t prefill any connection string to your database. That would make far too much sense. So if you don’t have the connection string ready here’s a quick way to do it:

An easy (ish) way to get SQL Server Connection string
First open up Windows Powershell. Let’s create a blank “.udl” file with this command:

     ni test.udl

Then open Windows Explorer in whatever folder you’re in:

    explorer .

Right click you’re new test.udl file and open Properties. Click into “Connection”.

Keep powershell open, but go back to SSMS and click “connect”, then if you’ve already connected to your database, it should pre-fill with the server name. Copy this and paste it in “server name” in the test.udl properties. Then enter your credentials/Windows NT integration for how you’ll authenticate to the db. Now select your database from the dropdown (this should prefill if the authentication and server name are correct). Finally click “Apply” and close. Back in PowerShell, you can now print it out and you should see the connection string:

    cat test.udl

And you can use this command to copy the second line (the connection string itself):

    Get-Content test.udl -TotalCount 3 | Select-Object -Last 1 | clip;

Now you can delete the .udl file if you want to keep things clean:

    rm test.udl

Finally, for our import, you can paste the connection string you’ve generated into the “Connection string” value in the wizard on the “Destination” page where we’re importing data.

Specify Table Copy or Query #

In this page the default “Copy data from one or more tables or views” will work for us if don’t need to modify the data.

Select Source Tables and Views #

Now we get to choose sources and destinations. On the left you’ll see the list of sheets in the excel file, on the right, you’ll see the list of tables in your database that the wizard will create/append to. You can edit the names of the tables. It will also try and guess the types from the excel file, but you can configure types with the “Edit mappings” button. It will default to appending to a table if the table exists and creating table if it doesn’t. In “Edit mappings” you can configure it to it to delete the existing rows.

Save and Run Package #

Here you can configure whether you want to run it immediately, save it as a .DTXS Integration Services Package file, or both. We want to run immediately so hit that then watch it run. If you do save it there are a many ways to run the .DTXS file for later (I tried to use the dtexec utility but even that wanted the 64 bit version of the Microsoft Access database engine 😣).

BULK INSERT Command #

The SQL BULK INSERT command is similar to the flat file insert, except that this time you’ll have to define your table before you create it. For this method, you will have to save your excel file as a CSV first. This is nice if you don’t want to use a GUI tool and want to just use SQL:

    -- Create the table first
    CREATE TABLE Countries(Id bigint,Country varchar(100),Population bigint ,President varchar(100));
    -- Then you can bulk insert
    BULK INSERT Countries
    FROM 'C:\Path\To\File\Countries.csv'
    WITH (
      FIELDTERMINATOR = ',',
      ROWTERMINATOR = '\n',
      FIRSTROW = 2
    );

Pretty nice that you can do it without having to click through all the steps of the “Insert Data…” method.

Using Excel Formulas to generate an SQL script #

Now we’re starting to get into more manual methods. These methods rely on generating an SQL script. This means the data upload time might be slower, as SQL server will have to parse all the commands you give it. If you have a lot of data this might take a long time. Creating a script does give you a bit more control to edit data as you go.

Lets say we have an excel table that looks like this:

id	country	population	leader
1	New Zealand	5000000	Jacinda Ardern
2	USA	300000000	Joe Biden
3	UK	30000000	Rishi Sunak

And we want to get this data into the database, we could first create our table:

    -- Create the table
    CREATE TABLE Countries(Id bigint,Country varchar(100),Population bigint ,President varchar(100));

Then use an excel formula to generate a string.

So we would start with the cell to the right of “Jacinda Ardern”, then type this formula:

    =CONCAT("INSERT INTO Countries (Id,Country,Population,Leader) VALUES (",A2,",'",B2,"',",C2,",'",D2,"');")

The string that this formula will generate should look like:

    INSERT INTO Countries (Id,Country,Population,Leader) VALUES (1,'New Zealand',5000000,'Jacinda Ardern');

And we can use the auto-formula function to pull down the formula to create each cell and make the script which will run through every line.

I’ve created a few of these in my time, and there are some gotchas to watch out for. It can be quite difficult to balance all the different quotes (the double quotes for the excel strings and the single quotes for the SQL strings). Also, if there are any single quotes in strings, this will cause issues (I would complain about Irish names here, but the Irish have been through enough). You can use the SUBSTITUTE function of excel to escape single quotes. So, if we were worried about our “leader” column having single quotes in the string, the formula would look like:

    =CONCAT("INSERT INTO Countries (Id,Country,Population,Leader) VALUES (",A2,",'",B2,"',",C2,",'",SUBSTITUTE(D2,"'","''"),"');")

Using awk to generate an SQL script #

I‘ve had a fascination for the old-school tool awk ever since listening to the Changelog interview with Brian Kernighan. Awk is a simple programming language for data processing which has been around since the 70s and gets included in bash scripting environments.

If you have linux or mac you will have awk ready to go by default. If you’re on windows, there are a myriad of ways to get a bash-like environment. I’d argue the simplest is git bash. If you’re using git, you might already have it installed on your machine.

A great way to use awk is to first save your file as a tab delimited text file. Let’s continue with the countries file above. First save that as countries.txt.

Then open git bash and navigate to the directory and type this command:

    awk -F'\t' $'{print}' countries.txt

This should print all the lines of the text file. An explanation:

awk is the bash command we’re using
-F'\t' is configuring it to use tabs as the delimiter
$``'``{print}``' is the awk script we’re running (in this case it will print every line)
countries.txt is the name of the file

Now if we were looking at our countries file we could use this script to generate the sql script we wanted:

    awk -F'\t' $'NR>1 {print "INSERT INTO countries (Id,Country,Population,Leader) VALUES ("$1",\'"$2"\',"$3",\'"$4"\')"}' countries.txt >> countries.sql

So, what is this new code doing?

NR>1 is skipping the header (only run this code if the row number is over 1)
Awk will automatically concatenate strings for you so we just throw them in a line.
We use \``' to escape the single quotes inside the awk command (this will only work in bash if you use a dollar sign at the start of the string)
>> countries.sql will command it to store the output in a new SQL file

Now, we’re going to have the same issue with this as we did with generating scripts in excel, that we don’t want single quotes to destroy our strings. We can use the sub function to prevent this from happening:

    awk -F'\t' $'NR>1 {sub(/\'/,"\'\'",$4); print "INSERT INTO countries (Id,Country,Population,Leader) VALUES ("$1",\'"$2"\',"$3",\'"$4"\');"}' countries.txt

Using node.js to generate an SQL script #

If you’re a normal person, you might want to use a more modern programming language than awk to generate a script. And we should never bet against JavaScript. We can use an npm package called [csv-parser](https://www.npmjs.com/package/csv-parser) to parse a CSV file and create an SQL script.

First, navigate to a folder, save your excel file as a .csv in it and use these commands to initialise npm and install csv-parser:

    npm init -y 
    npm install csv-parser

Now you can write a node script to create your SQL file. First, import fs and csv-parser

    const csv = require('csv-parser')
    const fs = require('fs')

Now you can declare an array to store the data in, pipe your file to the csv parser and loop trhough the array to print off a script:

    // script.js
    const data = [];
    fs.createReadStream('countries.csv')
      .pipe(csv())
      .on('data',(data)=>{results.push('data')})
      .on('end',()=>{
        let string = '';
        results.forEach(r=>{
          string += `INSERT INTO countries (Id,Country,Population,Leader) VALUES (${r['id']},'${r['country']}',${r['population']},'${r['leader']}');`
        })
    })

Then in git bash you can run your script and print it into an SQL file:

    node.exe script.js >> script.sql

Our JS script is less ugly compared to the awk script, but it’s still pretty ugly and took more set up. Still, Javascript is so ubiquitous that any developer would be able to do this.

Other methods #

Do you know quicker/better ways than these to get data from excel files into excel?

Get in touch on Bluesky or Mastodon and I’ll update this post with your method and give you a shout out.

What's in a PDF

2024-03-11T00:00:00Z

If you’ve been a computer person long enough, then you’ll eventually come to terms with the fact that all the files on a computer are made up of the same ingredients: lists of bytes. Whether you save a text file or a movie or an executable on your PC, the circuitry in the storage drive has no idea what it is. Even the operating system is just guessing based on context. If you give a file the correct extension, and it hasn’t been corrupted, the operating system knows what to do with it. Jim Nielsen has a good write up about the reusability of files on his blog.

Technically you could create every file you’ve ever used by writing out raw hexidecimals and using the bash xxd command to convert it into binary. If you’re on windows like me, you can use git bash. Let’s make an ASCII text file by writing some hexidecimal by smashing this into a terminal:

    echo '4865 6c6c 6f20 776f 726c 6421' | xxd -r -p > test.txt

If you open the new test.txt file with a text editor, you should see a text file saying “hello world!”. You can use this chart to write out any ASCII letter you want. Congratulations, you now have an extremely tedious way to create a text file. UTF-8 is a bit more complicated as it has almost every written language and every emoji.

However you can print a skull emoji like this:

     echo 'f0 9f 92 80' | xxd -r -p > test10.txt

These are ways to create text files. But binary files are more of a mystery. They are files whose secrets are known by the programs that read them, spec-writers who define them, and those who are willing to go through the pain-staking work of reverse-engineering them.

We’re going to take a look at a type of binary file: the pdf. Parts of the pdf format are actually understandable as ASCII text, and we’re going to write a PDF from scratch to see what's in there. Beware though, writing a pdf from scratch is a tough task and certainly isn't like writing a programming language designed to be easy to edit with a text editor.

Let’s take a look at the sections of a PDF:

Sections of a PDF #

There are always 4 sections to every pdf: first the header, then the body, then the cross reference table and finally the footer. We’re going to go in a weird order: The body first, then the header, then the cross reference table, and finally the footer. This is because the body is where the most interesting and important stuff is.

Body #

The body of a PDF holds the parts that will actually be visible in the PDF, as opposed to boilerplate and metadata. The PDF body holds the text, pages, images, fonts, annotations and form controls. There’s a whole smorgasbord of other stuff you can include too.

Different elements are structured as a hierarchical tree of “PDF objects”. PDF objects are written in a syntax like this:

    1 0 obj
    <<
      /Type /Catalog
      /Pages 2 0 R
    >>
    endobj

Let’s have a look at what we’ve written: First, we have a couple of numbers. The first number is a unique identifier for the object (in this case 1), the second is the version of the object (in this case 0). The version number is used for change tracking. We’re going to leave the version as 0 for all objects in this article. After these numbers we have obj. This tells the PDF reader we’re about to write a PDF object definition.

The next section is wrapped in a pair of less than << and greater than >> signs. This denotes that we are about to write a dictionary of key/value pairs. This is a bit like a json object or a C# dictionary or a java hashmap. The order does not matter here, just keys and values.

In the above example we have two keys: Type and Pages. In this case the value of type is Catalog. This is the type we use for the root object of a PDF document, as explained in the PDF standard. The value of the pages key is a reference to another object. It points to another object just like the one we’ve written. The two numbers after the /Pages key, 2 and 0 will match the unique identifier and version respectively. Then a capital R confirms that this is referencing another object.

Does any of the whitespace matter? The newlines and spaces? No. They will make a difference to the cross reference table later but that’s getting ahead of ourselves.

We finally close off our dictionary with two greater-than signs then complete our object with endobj.

Pages
We’ve actually written a great start to a pdf document already. But that /Pages key is referencing a PDF object that doesn’t exist. So now let’s write a “pages” object for it to point to:

    2 0 obj
    <<
     /Type /Pages
     /Count 1
     /Kids [
      3 0 R
     ]
    >>
    endobj

Again, we start with the unique identifier, version and obj keyword. The identifier and version have to match the values we wrote in the pages key in the catalog object.

Just like the catalog object, we say it’s of /Type /Pages. Then we have a new concept: an array! A list of things! In this case the /Kids key holds a list of references to page objects. We also include a count to say how many pages there will be (because there’s no way software would be able to figure that out from the number of references 🤔).

We’re only going to make one page but for demonstration purposes if we wanted three pages then we’d have something like this:

    /Count 3
    /Kids [
      3 0 R
      4 0 R
      5 0 R
     ]

The array is actually the 4th data type we’ve seen. These are the data types we’ve seen so far:

name objects (these are the keys with a forward slash before them)
dictionaries
numbers
arrays

The other pdf data types are:

booleans
strings
null
streams (this is where we’re going to put all our text and where you’d put your images)

These make up everything you can put into the body section of a pdf.
Page object
This is what our page object will look like:

    3 0 obj
    <<
     /Type /Page
     /Parent 2 0 R
     /MediaBox [
      0
      0
      612
      792
     ]
      /Resources <<
        /Font <<
         /Helv <<
          /Type /Font
          /Subtype /Type1
          /BaseFont /Helvetica
         >>
        >>
       >>
     /Contents [
      4 0 R
     ]
    >>
    endobj

Again we start with our object an ID and version, then inside the dictionary we give it a type. We also hold a reference to the parent (the pages object from earlier).

Now we actually have a definition of something visual: the size of the page with the /MediaBox array! First with the x and y coordinates of the lower left corner with 0, 0, then the coordinates of the upper right corner with 612, 792. PDF measures size from left to right, which makes sense to me being a native English reader, however it also measures from bottom to top.

We could stop here and we’d have a blank page (after adding the header, cross-reference table and trailer), but let’s put something on the page. To do this we’re first going to add a resources object which will describe the helvetica font as above.

Then we’ll continue down our chain of references, this time pointing to an object reference in the /Contents key. Let’s define this object now:

Contents object

Here’s what our next object looks like:

    4 0 obj
    <<
     /Length 53
    >>
    stream
    1 0 0 1 72 708 cm BT /Helv 12 Tf (Hello world!) Tj ET
    endstream
    endobj

It starts pretty normal, with the identifier and dictionary. Here we define a length. This is the length in bytes of the stream, which, if we keep it simple and use ASCII, will be one byte per character. This will count all the characters in between the newline character after the stream keyword and the newline character before the endstream keyword. These newline characters are not counted, but any spaces or tabs or other newline characters between these keywords will be counted in the length. The length does include any spaces or other whitespace.

As you can see, we throw these stream keywords after the dictionary. What is in this stream? First we write the scale and skew of the text. 1 0 0 1 is the section defining this and tells it not to rotate, skew, or scale the text in any way. Have a play with these numbers if you want to see the effect each of them on screwing around with text. I want normal text so I’m gonna leave it as 1 0 0 1.

Then we place the text: 72 cm left and 708 cm up (that’s centimetres if you’re an imperial measurement person). The BT keyword means we’re starting to define some text content. We specify the font, the font size and then use TF to finish of the font declaration (this is reverse Polish notation, where you write your two arguments, then name the function, we’re doing the same thing with the object identifier syntax).

Inside some parentheses we write the actual text we want on the page. Then writing Tj will pass this text to the Tj function, which will take the string and paint the glyphs. We write ET to end the text and we’re done!

This is the body of our pdf completed!

The header of the pdf is really easy. It basically always looks like this:

    %PDF-1.7
    %âãÏÓ

First we define that it’s a pdf, define the version of the spec, then there’s some gibberish letters to try and convince people not to do exactly what we’re doing right now and dig into the pdf. As far as I have read, you don’t actually have to put this exact set of gibberish in there, but these characters are standard so it’s what we’ll do.

Cross-reference table #

The cross-reference table is quite interesting. Now we’re really getting into something that looks like a binary file.

The cross reference table goes after the body, and is a table that software will use to hold the byte address of different objects inside the PDF. Assuming a PDF is all ASCII, it’s basically a count of how many characters (including newlines) before the start of each object.

This is how ours is going to look:

    xref
    0 5
    0000000000 65535 f 
    0000000015 00000 n 
    0000000068 00000 n 
    0000000133 00000 n
    0000000374 00000 n

Now we’re getting into some quite hardcore binary looking stuff.

First we start with the xref keyword to announce that we’re starting the cross-reference table. Then we have two numbers. The first is the “generation number” - used for versioning, we’re just going to leave it as zero. The second is the number of objects in our cross reference table. We only made 4 pdf objects but I wrote a 5. What gives?

We always add one extra placeholder object to the start of the xref table. It’s a way to indicate the start of the table.

After declaring the generation and number of objects in the table we start the table. Each row of the table has 3 columns:

The byte-offset from the start of the file to the object
The generation number of the object
Either an n to say the object is in use or a f to say the object is free and doesn’t actually matter. I’m not sure what the n actually stands for, but let’s pretend it stands for “now-in-use” or “not-retired”

It’s important that you left-pad the numbers with zeroes to keep all the items in the table the exact length above.

Here’s our first placeholder object, which will always look like this:

    0000000000 65535 f

The second object points to the object in our pdf with an ID of 1:

    0000000015 00000 n

The start of the first pdf object (our catalog object, specifically at the point we gave it an ID of 1) is 17 characters from the start of the file, hence the first number in the xref table is 15. This is the byte-offset of the object. We aren’t bothering with generations, so the second number is 0. Finally, this object is in use, so we add an n to indicate it’s in-use. We need to do this with all the objects in the pdf.

Getting the byte offset of a PDF section #

You can use grep to get the byte offset of an object:

    grep --byte-offset --text $'1 0 obj' mypdf.pdf

Where you replace the number “1” in the code above with the ID of the object you want the offset for.

Trailer #

We’re almost there! We’re at the last section of our PDF file.

This is what our trailer will look like:

    trailer
    <<
     /Root 1 0 R
     /Size 5
    >>
    startxref
    491
    %%EOF

We announce it’s a trailer, then we enter another dictionary with a paid of less-than signs.

This dictionary holds 2 keys:/Root and /Size. The root is a reference to the catalog object. The value of size is the number of rows in the cross-reference table.

Finally, after we’ve closed off the dictionary, we describe the byte-offset of the start of the cross-reference table. We don’t actually want the “xref” keyword but the offset of the first object of the table so using the grep script from before to grab byte-offsets we can use:

    grep --byte-offset --text '0000000000' mypdf.pdf

Then, finally to confirm we’re done we add an end of file marker with a couple of percent signs:

    %%EOF

The whole file #

This is what the entire pdf should look like. Save this in a text file as a pdf, and if nothing’s gone weird with encoding, you should have a fully functional pdf:

    %PDF-1.7
    %âãÏÓ
    1 0 obj
    <<
      /Type /Catalog
      /Pages 2 0 R
    >>
    endobj
    2 0 obj
    <<
     /Type /Pages
     /Count 1
     /Kids [
      3 0 R
     ]
    >>
    endobj
    3 0 obj
    <<
     /Type /Page
     /Parent 2 0 R
     /MediaBox [
      0
      0
      612
      792
     ]
      /Resources <<
        /Font <<
         /Helv <<
          /Type /Font
          /Subtype /Type1
          /BaseFont /Helvetica
         >>
        >>
       >>
     /Contents [
      4 0 R
     ]
    
    >>
    endobj
    4 0 obj
    <<
     /Length 53
    >>
    stream
    1 0 0 1 72 708 cm BT /Helv 12 Tf (Hello world!) Tj ET
    endstream
    endobj
    xref
    0 5
    0000000000 65535 f 
    0000000015 00000 n 
    0000000068 00000 n 
    0000000133 00000 n
    0000000374 00000 n
    trailer
    <<
     /Root 1 0 R
     /Size 5
    >>
    startxref
    491
    %%EOF

I don’t know about other editors, but in vim you can force it to treat the text as ASCII to make sure this saves with correct byte offsets by using these settings in vim:

    set encoding=latin1
    set isprint=
    set display+=uhex

So after you set these settings in vim, you can paste the above file then save, then you should have a working PDF file. Hooray!

A good way to absolutely verify things are correct is to use the GitHub - pdfcpu/pdfcpu: A PDF processor written in Go to verify your PDF with the “pdfcpu validate mypdf.pdf” command.

Where to go from here #

Daniel Warren wrote a similar blog post to this back in 2010 (14 years ago 👴). You should check out.
You can get the full PDF standard from adobe

OIDC Authentication in server-side Blazor apps

2024-07-17T00:00:00Z

Today, we're going to create a server-side Blazor app that uses Azure OIDC to authenticate. You’ll need the DotNetCore SDK. I’m going to be using DotNetCore 6 - the oldest possible version that's still supported.

First I’m going to set things up. I’m not going to go into a whole lot of detail until we get into authorization section, so if this is your first time creating a DotNetCore app, you might want to find a more introductory post. I'll also be using the command line and VSCode, so this may not be for Visual Studio fans.

Let’s start by creating a new server-side blazor app:

dotnet new blazorserver -o BlazorServerApp

Now cd into the app, then there’s a command you might need to run to get the nuget install going:

    cd BlazorServerApp
    dotnet nuget add source --name nuget.org https://api.nuget.org/v3/index.json

Now let’s run the app. I like to use the watch command so the server will reload when I make a change:

    dotnet watch -- run

Head to https://localhost:7092/ (or whichever port it has decided to host from) and you should be looking at a brand new server-side blazor app.

Setting up Azure as an OIDC provider #

Now let’s create an OIDC server with Azure. Basically we’re going to follow this blogpost’s instructions.

We’re going to create an app registration. You’ll need to have some kind of elevated permissions to do this.

Head to the Azure portal, then we’re going to go to Microsoft Entra app registration.

Give your app a name - I’m going to call it “blazorserverapp”. After it’s created, head to “clients and secrets” and create a new client secret. Grab the secret and the client ID and keep these in a text file for now.

You’ll also need to add a redirect URL so that Azure knows where your app is based and isn’t going to send a load of private information to someone else’s domain. Click into “Authentication” then under “redirect URIs”, add https://localhost:7102/signin-oidc. We want our site’s address with /signin-oidc at the end. You can add multiple URLs for development and production if you need.

Adding OpenIDConnect to your Blazor app #

Now, server-side blazor doesn't link super well with the OpenIDConnect package. This package expects that client and server will be communicating using HTTP, which isn't the protocol that server-side blazor uses. Instead, Server-side Blazor uses a protocol called SignalR. Don't fear though! We're going to (partly) follow the methods of this stack overflow answer to get around these issues.

First let’s install the OpenIdConnect package:

    dotnet add package Microsoft.AspNetCore.Authentication.OpenIdConnect

Running DotNet 6 means I needed to add this to install an older version:

    dotnet add package Microsoft.AspNetCore.Authentication.OpenIdConnect --version 6.0.2

Let’s start by updating our Startup.cs file to configure our app to use the OpenIdConnect authentication package. First add the using statement at the top:

    using Microsoft.AspNetCore.Authentication.OpenIdConnect;

Then anywhere before the app = builder.Build(); statement add in this to register the authentication:

    builder.Services.AddAuthentication(options =>
    {
        options.DefaultScheme = "Cookies";
        options.DefaultChallengeScheme = "oidc";
    })
    .AddCookie("Cookies")
    .AddOpenIdConnect("oidc", options =>
    {
        options.Authority = "https://login.microsoftonline.com/mytenant/v2.0/";
        options.ClientId = "my client id"; 
        options.ClientSecret = "my client secret";
    });

Then after the app = builder.Build(); statement, and before app.UseRouting();, we tell it to use authentication in the middleware pipeline:

    app.UseAuthentication();

This sets up the authentication. Now when we we start the app the OpenIdConnect middleware will be listening on the /signin-oidc endpoint for when Azure sends back user information.

There will still be no difference to the app until we start to configure which pages are public and which pages only authenticated users can see.

Hiding pages behind authentication #

We're going to update the App.razor file to check whether the user is logged in and redirect them to a login page if not:

    <!-- App.razor -->
    @inject NavigationManager NavigationManager
    <CascadingAuthenticationState>
    <Router AppAssembly="@typeof(Program).Assembly">
        <Found Context="routeData">
            <AuthorizeRouteView RouteData="@routeData" DefaultLayout="@typeof(MainLayout)">
                <NotAuthorized>
                    @{
                        
                        var returnUrl = NavigationManager.ToBaseRelativePath(NavigationManager.Uri);
                        NavigationManager.NavigateTo($"login?redirectUri={returnUrl}", forceLoad: true);
                    }
                </NotAuthorized>
                <Authorizing>
                    Wait...
                </Authorizing>
            </AuthorizeRouteView>
        </Found>
        <NotFound>
            <LayoutView Layout="@typeof(MainLayout)">
                <p>Sorry, there's nothing at this address.</p>
            </LayoutView>
        </NotFound>
    </Router>
    </CascadingAuthenticationState>

This will only redirect for pages where there is an @attribute [Authorize] on the page. All pages without this attribute will remain open to the public. Let’s update our index.razor page to the following:

    @page "/"
    @using Microsoft.AspNetCore.Components.Authorization
    @attribute [Authorize]
    
    <PageTitle>Index</PageTitle>
    
    <h1>Hello, world!</h1>
    Welcome to your new app.
    <SurveyPrompt Title="How is Blazor working for you?" />

If you run the app and head to the home page now, you’ll get redirected to a login URL in our site which does not exist.

Redirecting users to the OIDC provider #

Let’s create the endpoint the unauthenticated user will get redirected to. We use this endpoint to forward the user to our OIDC provider. Write a login.cshtml file in the Pages folder of the app (cshtml files get server-rendered and aren’t part of the blazor app):

    @page
    <!-- Login.cshtml -->
    @model LoginModel

That’s all we need in the login.cshtml file, but now we add the model file (Login.cshtml.cs) in the same directory to define the logic after a request:

    // Login.cshtml.cs
    using Microsoft.AspNetCore.Authentication;
    using Microsoft.AspNetCore.Mvc.RazorPages;
    
    public class LoginModel : PageModel
    {
        public async Task OnGet(string redirectUri)
        {
            if (!User.Identity.IsAuthenticated){
                await HttpContext.ChallengeAsync("oidc", new 
                    AuthenticationProperties { RedirectUri = redirectUri } );
            }
            else{
                // redirects user to redirectUri if they are already logged in
                if (redirectUri == null)
                {
                    redirectUri = "/";
                }
                Response.Redirect(redirectUri);
            }
        }  
    }

The HttpContext.ChallengeAsync function is the part which does the work, redirecting the user to Azure if they are not logged on.

Now after adding this and restarting your app, you should get redirected to Azure to login, then get redirected back to your application again. If you go through this process then look in the “Application” section of your browser’s dev tools you should notice that there are new cookies associated with your app. This is where your session is being kept track of.

Using claims from OIDC in your app #

How do we get access to user data we’ve received from Azure?

We can use blazor's AuthenticationStateProvider class. This can be injected into our blazor components to get accessnto the logged in user. This class will allow us to get a handle on all the claims we’re getting back from Azure. To demostrate this, update your index.razor file to look like this:

    @page "/"
    @using Microsoft.AspNetCore.Components.Authorization
    @using System.Security.Claims
    @attribute [Authorize]
    
    <PageTitle>Index</PageTitle>
    <h1>Hello, @User.Identity.Name!</h1>
    <ul>
        @foreach (var claim in User.Claims)
        {
            <li>
                <strong>@claim.Type:</strong> @claim.Value
            </li>
        }
    </ul>
    
    Welcome to your new app.
    <SurveyPrompt Title="How is Blazor working for you?" />
    @code{
      @inject AuthenticationStateProvider AuthenticationStateProvider
      
      @code {
          protected ClaimsPrincipal User;
          protected override async Task OnInitializedAsync()
          {
              var authState = await AuthenticationStateProvider.GetAuthenticationStateAsync();
              User = authState.User;
          }
      }
    }

This will say hello to the user, pulling their name from the user identity data we get back from Azure, and will also list all of the other claims it receives. An important claim is preferred_username, which in my case was my email address - this could be used to link the logged in user to data in your database.

Wrapping up #

This is a small start, but from here the sky is the limit. You don’t have to manage authentication yourself. Your users don’t have to create another password. But after they log in, you know who they are and you can create logic around what they can see and do.

Where to from here? You can look into configuring Azure to send different claims. These can be used to decide what the user is allowed to see; You can implement claims-based authorization on your pages. Or if there is data in your application that should remain hidden, you can write logic to hide it from users without certain claims.

If you’re a blazor expert and you have more tips about OIDC authentication to server-side blazor apps, let me know! I’m on Mastodon and Bluesky.

Other resources #

OIDC Authentication in ASP.NET Core apps explained

2024-10-14T00:00:00Z

In the times that I’ve had to implement Single Sign On authentication for ASP.NET Core apps, I’m always trying to do it as quickly as possible. I usually get it done in a single-minded period filled with anger at documentation and bewilderment around what I’ve configured incorrectly, then eventually I’ll achieve a signed in user (yay). The information online about OpenIDConnect in ASP.NET Core is either:

2 lines telling you to call a couple of methods with no context or explanation; or
A meandering novel filled with technical jargon

This post is here to help you avoid what I’ve been through. It’s here to help you reason about what’s going on when you use OIDC authentication in ASP.NET core.

What is OIDC? #

OIDC, or OpenIDConnect is a protocol for Single Sign On. It allows everyone in your business to have a username and password with the OIDC provider, and they don’t need to remember a separate username and password for your special snowflake app (and the other 40 special snowflake apps they have to use every day at the business).

Some definitions:

The OIDC provider is a 3rd party - eg. Microsoft Azure, Google, Okta etc
The OIDC client is your app

OIDC is built on top of another protocol called OAuth. OAuth is also for communication between apps, but it’s for a user to be able to give your app permissions to access something in another app, whereas OIDC is designed to tell your app who the user is. OAuth is about authorization, OIDC is about authentication. You can find more about the difference on Nat Sakimura’s blog.

The OIDC provider tells our app who the user is by sending a post request to our app with url-encoded form data. This data will include an ID token, which is a JWT: a Base64 encoded string of JSON providing information about the user. You can read more on the Microsoft documentation about what the ID token format looks like.

Let’s create a web app that uses OIDC Single Sign On. I’m going to first provide instructions to create the app, then I’ll go deeper into what we’ve done:

Setting up Azure as an OIDC provider #

First set up an OIDC provider. This isn’t the focus of this post but in my last post, I wrote a bit about how to set up azure as your OIDC provider. We basically just follow this handy tutorial from the Microsoft docs. After you’ve set it up, you’ll need your tenant ID, client ID, and client secret.

Scaffolding a web app with OIDC #

Let’s scaffold an ASP.NET core web app. We’ll create an MVC application. To do this, you’ll need the DotNetCore SDK installed. Navigate to a new project folder and run this in the command line:

    dotnet new mvc

Then you should be able to start your app:

    dotnet run

This will print off a URL where you will be able to see your app in action. You’ll need to update whatever URL you see here with your OIDC provider - with signin-oidc as the relative path. This is so the provider knows where to redirect back to on sign in. Now let’s install the OpenIDConnect package:

    dotnet add package Microsoft.AspNetCore.Authentication.OpenIdConnect

We'll add a using statement to the top of our Program.cs file to import this package. We’re also going to import the cookies authentication namespace (you don’t need to install any more packages to get this):

    // Program.cs
    using Microsoft.AspNetCore.Authentication.OpenIdConnect;
    using Microsoft.AspNetCore.Authentication.Cookies;

Then anywhere before the app = builder.Build();statement in Program.cs, add the following lines of code to register our authentication. If you’re using Azure Active Directory, this where you add your tenant ID, your client ID and your client secret:

    // Program.cs
        builder.Services.AddAuthentication(options =>
        {
            options.DefaultScheme = CookieAuthenticationDefaults.AuthenticationScheme;
            options.DefaultChallengeScheme = OpenIdConnectDefaults.AuthenticationScheme;
        })
        .AddCookie()
        .AddOpenIdConnect(options =>
        {    
            // Replace mytenantid with your tenant ID
            options.Authority = "https://login.microsoftonline.com/mytenantid/v2.0/";
    
            // Replace my client id with your client ID
            options.ClientId = "my client id"; 
    
            // Replace my client secret with your client secret
            // This should not be committed to source and would be better living in an
            // appsettings.json file
            options.ClientSecret = "my client secret";
        });

Now, after the app.UseStaticFiles(); statement, and before app.UseRouting();, add the following line to add authentication to the middleware pipeline. The position of this method call matters:

    // Program.cs
    app.UseAuthentication();

Now we’re mostly set up, let’s hide a page from non-authenticated users. Head to your HomeController.cs file. First we’ll add a using statement at the top to pull in the Authorization namespace:

using Microsoft.AspNetCore.Authorization;

Now add an [authorize] attribute above the Index method:

    // HomeController.cs
       [Authorize]
        public IActionResult Index()
        {
            return View();
        }

When you hit the index page of your application, you should now be redirected to a Single Sign On page from Microsoft. Hooray!

We can update the controller and the index page to show data we’ve received back from Azure about who the user is. Here’s how we change our HomeController’s Index method:

    //HomeController.cs
    
        [Authorize]
        public IActionResult Index()
        {
            ViewData["name"] = HttpContext.User.Claims.First(c=>c.Type=="http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name").Value;
            return View();
        }

And here’s how we change our Index.cshtml page:

    //Index.cshtml
    @{
        ViewData["Title"] = "Home Page";
    }
    <div class="text-center">
        <h1 class="display-4">Welcome @(ViewData["name"] )</h1>
        <p>Learn about <a href="https://learn.microsoft.com/aspnet/core">building Web apps with ASP.NET Core</a>.</p>
    </div>

Now when you hit your index page and sign in, you should see your name in big letters. We’re going to dive into what is happening under the hood to make this work.

What have we done? #

We haven’t added many new lines to our application, but we’ve gone from an app that has no idea who the useris, to one that integrates with Microsoft to get information about the user and can lock unauthenticated usersout on a per-controller-method basis. That’s a lot of heavy lifting we’re getting from the framework and theOIDC library, so let’s get into how this actually works:

The OpenIDConnect library and ASP.NET Core Cookies Authentication #

One of our first steps was to install the Microsoft.AspNetCore.Authentication.OpenIdConnect package. This package is doing the heavy lifting to get OIDC working. You can find the code for this package deep within the AspNetCore source code on github. We’re also using Microsoft’s cookie authentication package, which you can find in the AspNetCore github repository. The cookies package comes included in with the ASP.NET core framework so we don’t have to install anything from nuget to use it. What are these libraries doing?

Registering Authentication #

First we registered our authentication schemes by calling AddAuthentication. Authentication schemes are different sets of logic for authenticating the user. ASP.NET Core uses a string-based registration system for keeping track of authentication schemes, where it stores a dictionary of string-to-AuthetnicationScheme mappings. There’s a good writeup about Authentication Schemes on Matteo Contrini’s blog. In our app above we passed it strings for our authentication schemes by setting properties of the [AuthenticationOptions](https://learn.microsoft.com/en-us/dotnet/api/microsoft.aspnetcore.authentication.authenticationoptions?view=aspnetcore-8.0) class. We set the default scheme to be cookies and our default challenge scheme to be OpenIDConnect.

In AddAuthentication when we set the schemes, we were actually just passing in strings, so we could change the addAuthentication lines in our code above to:

        builder.Services.AddAuthentication(options =>
        {
            options.DefaultScheme = "Cookies";
            options.DefaultChallengeScheme = "OpenIdConnect";
        })

And this would do the same thing as the earlier code we wrote. The issue might be that the two strings "``Cookies``" and "``OpenIdConnect``" for the packages default scheme names might change, so it’s better practice to import the default names from the package defaults.

Why are there two schemes? One for cookies and one for OIDC? The reason is that most of the work of handling authentication can be done with cookies authentication, but we want the challenge (the point we ask the user to login with username/email and password) to be done by OpenIDConnect. The OIDC package is a specific type of scheme called a RemoteAuthenticationScheme.

This type of scheme is for third-party single sign on, but only handles the challenge, and requires another scheme for persisting the user after they’ve logged in. We can keep information about the logged in user with cookie authentication, which will attach a cookie to the user’s browser to store who they are. This cookie API will also handle making sure that the user isn’t tampering with their cookie to pretend to be someone else.

Passing a couple of strings into the app builder isn’t enough to fully register our authentication. We also have to execute a couple of extension methods provided by the libraries. First we run the AddCookie method, then the AddOpenIdConnect method on the returned authentication builder (passing OpenIdConnect the details for our open ID connect setup).

Under the hood these extension methods will set up their configuration, and call methods to register the app logic for their authentication schemes. This is done when the AddScheme and AddRemoteScheme methods get called on the AuthenticationBuilder Class. These methods each register an authentication handler class.

Authentication Handlers #

Authentication handlers are where the actual logic of the authentication happens. You can find a good post about them on joints westlins blog.

Authentication handlers are used to check whether the user is logged in. They need to implement one method: HandleAuthenticateAsync This method will get called for every request - we triggered it by adding the [authorize] attribute to our controller. Authentication handlers have access to all the HTTP request information, which is how the OIDC library can access the JWT data sent by our OIDC provider.

When the middleware pipeline gets to a page/endpoint that requires authentication, the endpoint will check if the user is logged in based off the result HandleAuthenticateAsync returns. This method needs to either return Success, Failure or NoResult based on the HTTP request. What is the difference between failure and no result? If there are multiple types of authentication registered to your app, then passing NoResult will pass it to another one of the authentication handlers. Failure will mean that our handler should handle the request, but the check fails. If every authentication scheme returns NoResult then the user also won’t be able to access the page.

Inside the OpenIDConnectHandler, the HandleRemoteAuthenticateAsync method will listen for POST requests and will check whether they look like an incoming OIDC message. If so, it will parse the JWT, verify that it’s from the OIDC provider, and verify it’s responding to a request our app sent. The OpenIDConnectOptions class inherits from the RemoteAuthenticateOptions class, which has a property called [SignInScheme](https://github.com/dotnet/aspnetcore/blob/f14f99eb634dc5a3ce8d5e11c7522577e5473e90/src/Security/Authentication/Core/src/RemoteAuthenticationOptions.cs#L112). This SignInScheme is the scheme that will persist the user (it will default to the default authentication scheme). In our case: the Cookies scheme. This property is what allows our two schemes to connect. After the OpenIDConnectHandler finds the user has successfully logged in, the RemoteAuthenticationHandler parent class will call Context.SignInAsync, passing in the SignInScheme to tell the Cookies authentication to store the user.

Forwarding the challenge to the OIDC provider #

We know how the user gets authenticated after the OIDC provider gets back to us, but how does the OIDC provider know we have a user who needs to log in?

To ask for this, we send a Sign-In Request to our authentication provider. To do this, we need to send a 302 redirect to the user sending them to the provider’s authorization endpoint. In our case with Azure, the authorization endpoint is the https://login.microsoftonline.com/mytenantid/v2.0 URL we specified as the Authority property in our Program.cs file. If you’re using another provider, the OIDC spec defines that there should be a metadata file describing what the authorization endpoint is. When we send the request to the authority, we also provide a redirect URL for the provider to send the user back to us, and our client ID in the GET request parameters (among a few other things).

I say “we” do this, but it’s all handled by the OpenIdConnectHandler class. AuthenticationHandler classes have a method you can override called HandleChallengeAsync, which gets called when there is a 401 Unauthorized HTTP response code later in the middleware pipeline. Our [Authorize] attribute on our controller method will trigger this when the user isn’t logged in. The HandleChallengeAsync method of the OpenIdConnectHandler will change the response to a 302 Redirect and send the user to the OIDC provider. The default redirect URL to send the user back to is signin-oidc.

Middleware pipeline #

We’ve registered what authentication we’re using. However, we need to put our authentication into our middleware pipeline to tell ASP.NET core what stage in the pipeline the authentication should go. In the example above, we registered the middleware after app.UseStaticFiles(); statement, and before app.UseRouting();.

The order matters. This is because the order of the methods you call in this part of the program.cs file decide the order that each piece of middleware gets run If we wanted to prevent unauthenticated users from accessing our static files we could move the useAuthenticate call to before UseStaticFiles call (we’d also need to make an update to the UseStaticFiles options):

    app.UseAuthentication();
    app.UseStaticFiles(
        new StaticFileOptions
        {
            OnPrepareResponse = ctx =>
            {
                if (!ctx.Context.User.Identity.IsAuthenticated)
                {
                    ctx.Context.Response.StatusCode = (int)HttpStatusCode.Unauthorized;
                }
            }
        }
    );

And this would mean only authenticated users would be able to access the static files (CSS, JS, images etc…) in our site.

This also means that any middleware which might have tampered with HTTP requests before they get to the authentication handlers may affect how our authorization handlers respond to the request.

Claims #

Once a user is logged in, we can access their information in controllers and middleware. The user's details are available through the HttpContext.User object, which is of type ClaimsPrincipal. This is a class which holds a list of claims. Claims in ASP.NET Core are string key-value pairs that store information about the user. Integrating with Azure, you’ll get back something like this (among other claims):

    Claim Name: name
    Claim Value: Fred Flinstone
    
    Claim Name: http://schemas.microsoft.com/identity/claims/objectidentifier
    Claim Value: A-GUID-Here-asdf
    
    Claim Name: preferred_username
    Claim Value: FredFlinstone@gmail.com
    
    Claim Name:http://schemas.xmlsoap.org/ws/2005/05/identity/claims/nameidentifier
    Claim Value: randomstring
    
    Claim Name:ttp://schemas.microsoft.com/identity/claims/tenantid
    Claim Value: A-GUID-Here-asdf

As you can see, you can get the name of a user, and their preferred username.

Matching user with database data #

Often you’ll have a bunch of data associated with your single-sign-on user stored in your application’s database. You don’t want to hit your database every time the user makes a request, so it’d be good to have your important database info in the ClaimsPrincipal. The ideal way to do this is using the OnTicketReceived event. This is an event which will fire after the user has successfully logged in. We can add the claims that we want here, which will cache information about the user. Here’s an example of how to do this:

      .AddOpenIdConnect(
            "oidc",
            options =>
            {
            /* 
             * Insert Tenant id, client id, ClientSecret code here 
             */
                options.Events = new OpenIdConnectEvents
                {
                    OnTicketReceived = e =>
                    {
                        ClaimsIdentity claimsIdentity = new ClaimsIdentity();
                        /* Hit database and grab information about user here */
                        claimsIdentity.AddClaim(new Claim("myNewClaim", "myClaimValue"));
                        e.Principal.AddIdentity(claimsIdentity);
                        return Task.CompletedTask;
                    }
                };
            }
        );

In the OnTicketReceived event, you could match the user with the preferred_username, or email address, or user ID, with a table in your database, then alter their claims based on what you get back.

Wrapping up #

Hopefully this helps you with adding OIDC to your ASP.NET Core apps. We looked at configuration, adding middleware, authentication schemes, user claims, and authentication events. OIDC can save you and your users from the frustration of managing multiple logins, but the amount of magic in ASP.NET Core can mean some of that frustration gets passed to you. Maybe a little less after this guide! At the very least, if you are able to set up OIDC authentication with your app, you’ll thank yourself for not having to deal with the stress of storing passwords and dealing with forgotten passwords. Anything is better than that!

Speed up Power BI merges with record lookups

2024-11-14T00:00:00Z

You’re in Power BI. You’re in the Power Query editor getting data. Imagine you’ve got a CSV like this:

id	name	salary
EMP-1	Johnny Bravo	10000
EMP-2	Donald Duck	20000
EMP-3	Finn the Human	15000
EMP-4	Jake the Dog	5000

And another CSV like this:

id	country
EMP-1	New Zealand
EMP-2	China
EMP-3	United Kingdom
EMP-4	Australia

And you want to combine the two to get:

id	name	salary	country
EMP-1	Johnny Bravo	10000	New Zealand
EMP-2	Donald Duck	20000	China
EMP-3	Finn the Human	15000	United Kingdom
EMP-4	Jake the Dog	5000	Australia

The first thing I’d usually do is jump to is using the “Combine” function and merge the two tables together on the “id” column. After this, I could expand the resulting merged table to get the country value in my employees table.

This is how this might look:

Getting the employees table:

    /* table name: employees */
    let
        Source = Csv.Document(File.Contents("C:\path\to\employees.csv"),[Delimiter=",", Columns=3, QuoteStyle=QuoteStyle.None]),
        #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
        #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{ {"id", type text}, {"name", type text}, {"salary", Int64.Type}})
    in
        #"Changed Type"

Getting the countries table:

    /* table name: employeecountries */
    let
        Source = Csv.Document(File.Contents("C:\path\to\employeecountries.csv"),[Delimiter=",", Columns=2, QuoteStyle=QuoteStyle.None]),
        #"Promoted Headers" = Table.PromoteHeaders(Source, [PromoteAllScalars=true]),
        #"Changed Type" = Table.TransformColumnTypes(#"Promoted Headers",{ {"id", type text}, {"country", type text}})
    in
        #"Changed Type"

And I’m going to make a third table which combines these with a merge:

    /* table name: merge */
    let
        Source = #"employees",
        #"Merged Queries" = Table.NestedJoin(Source, {"id"}, employeecountries, {"id"}, "employeecountries", JoinKind.LeftOuter),
        #"Expanded employeecountries" = Table.ExpandTableColumn(#"Merged Queries", "employeecountries", {"country"}, {"country"})
    
        
    in
        #"Expanded employeecountries"

This works great for some small CSV files on your local machine, but if you’ve got tables with hundreds of thousands of rows, and your sources are from across the internet, you’re going to hit some performance problems.

A faster way #

First we want to convert the countries into a record. As far as I can tell there’s no way to do this in the GUI, so you’re going to have to jump into the “advanced query” section to smash in this code:

    /* table name: employeecountriesrecord */
    let
        Source = #"employeecountries",
        #"Renamed Columns" = Table.RenameColumns(Source,{ {"id", "Name"}, {"country", "Value"}}),
        ToRecord = Record.FromTable(#"Renamed Columns" )
    in
       ToRecord

After you’ve added this in, it’s important to right-click this query and switch off “enable load” otherwise Power BI will automatically try to turn this query back into a table.

Now to grab the data. You can use the Record.FieldOrDefault function to do a lookup on the related table. This will default to null if it can’t find the key in the lookup table. If you want it to fail when there is no matching key, you can use ‘Record.Field’ instead.

    /* table name: lookup */
    let
        Source = #"employees",
        #"Added Custom" = Table.AddColumn(Source, "country", each Record.FieldOrDefault(employeecountriesrecord,[id]))
    in
        #"Added Custom"

The bigger your data is, the bigger the difference this will make, as a merge will be processed in O^n2 time (it will exponentially increase with the number of rows), whereas a lookup will not increase with the size of the data.

For demonstration, you can use git bash to automatically build larger tables with these scripts (okay these were written by chat GPT, but I used my extensive bash knowledge to audit them 👀).

    #!/bin/bash
    
    # Creates employees table
    
    # Set the output file and the number of rows
    OUTPUT_FILE="employees2.csv"
    NUM_ROWS=100000  # Adjust this for the number of rows you want
    
    # Define some sample names
    NAMES=("Johnny Bravo" "Donald Duck" "Finn the Human" "Jake the Dog" "Mickey Mouse" "SpongeBob SquarePants" "Rick Sanchez" "Morty Smith" "Homer Simpson" "Peter Griffin")
    
    # Create the CSV header
    echo "id,name,salary" > "$OUTPUT_FILE"
    
    # Generate random data for each row
    for i in $(seq 1 $NUM_ROWS); do
        # Unique employee ID
        ID="EMP-$i"
        # Random name from the list
        NAME="${NAMES[$RANDOM % ${#NAMES[@]}]}"
        # Random salary between 5000 and 30000
        SALARY=$(( (RANDOM % 25000) + 5000 ))
    
        # Write row to file
        echo "$ID,$NAME,$SALARY" >> "$OUTPUT_FILE"
    done
    
    echo "Generated $NUM_ROWS rows in $OUTPUT_FILE"

And this one:

    #!/bin/bash
    
    #Creates countries lookup table
    
    # Set the output file and the number of rows
    OUTPUT_FILE="employeecountries2.csv"
    NUM_ROWS=100000  # Adjust this for the number of rows you want
    
    # Define a list of sample countries
    COUNTRIES=("New Zealand" "China" "United Kingdom" "Australia" "United States" "Canada" "Germany" "France" "Japan" "India")
    
    # Create the CSV header
    echo "id,country" > "$OUTPUT_FILE"
    
    # Generate random data for each row
    for i in $(seq 1 $NUM_ROWS); do
        # Unique employee ID
        ID="EMP-$i"
        # Random country from the list
        COUNTRY="${COUNTRIES[$RANDOM % ${#COUNTRIES[@]}]}"
    
        # Write row to file
        echo "$ID,$COUNTRY" >> "$OUTPUT_FILE"
    done
    
    echo "Generated $NUM_ROWS rows in $OUTPUT_FILE"

Now we can add some more queries to check how long these are taking using Chris Webb’s check for how long our queries take:

    let
    StartTime = DateTime.LocalNow(),
    Source = if StartTime<>null
    then
     #"merge"
    else
    null,
    //insert all other steps here
    NumberOfRows = Number.ToText(Table.RowCount(Source)),
    EndTime = DateTime.LocalNow(),
    Output = "Query returned " & NumberOfRows & " rows and took " & Duration.ToText(EndTime - StartTime),
    end = Table.FromList({Output})
    in
    end

and:

    let
    StartTime = DateTime.LocalNow(),
    Source = if StartTime<>null
    then
     #"lookup"
    else
    null,
    //insert all other steps here
    NumberOfRows = Number.ToText(Table.RowCount(Source)),
    EndTime = DateTime.LocalNow(),
    Output = "Query returned " & NumberOfRows & " rows and took " & Duration.ToText(EndTime - StartTime),
    end = Table.FromList({Output})
    in
    end

At the end of this, my lookup takes 40% of the time of the merge with 10,000 row files. This gap will only increase as the number of rows increase and as network latency slows your query down. Keep this method in your toolbox, because it’s an important feature to speed up queries.

Is that really a merge? #

Now if you’ve been scrutinising this, you’ll notice that this isn’t a real merge! If you have a one-to-many or a many-to-many relationship between your two tables, the code above won’t help you.

What I mean by this, is that if you had employees who worked in more than one country and you had 2 rows in the country table for a single employee, and wanted to end up with 2 rows after your merge in your output. Using record lookups won’t do this for you and you should go back to using the merge queries function.

The thing is, most of the time I find myself reaching for merge function, I don’t want to end up with more rows. In fact, I usually want to actively avoid this as it might cause bugs with the resulting duplicates. Using a record lookup will actively enforce that you end up with the same number of rows you started with, with no duplicate IDs.

If you have more power bi performance tips, or you want to complain about this post, hit me up on Blue Sky.

Snippets everywhere on Windows with autohotkey

2024-12-31T00:00:00Z

One of the big problems with doing anything on computers is getting the information from your brain into the computer. This is opposed to the other (bigger) problem of solving/understanding the problem in your brain. As developers, the biggest thing that separates us from non-developers is that we communicate to the machine through text. Like Steve Yegge said: Text is king.

Getting text into the computer is an important mechanical process and we want it to happen as quickly as possible. There are a lot of methods to speed this up; learning to touch type, learning vim (or any editor tooling), using large-language-model autocomplete (which can save a lot of typing regardless of the quality it produces). I‘ve been looking at an underrated productivity boon: having a bunch of text snippets at your disposal. There are so many cases where we are typing the same thing; for loops, React components, common SQL queries. The list goes on.

Any code editor worth its salt will have a snippet feature, but I found that I needed some of the same text snippets in programs outside my text editor. For instance; what if I need the same snippet in SSMS to query a database and in VSCode for a migration script? What if I need to use a JavaScript snippet in the browser devtools console as well as in my source code?

If you’re using a Mac, you can laugh your way over to Raycast and use their Snippets feature. Unfortunately for us Windows users, we don’t have Raycast (yet).

However, what we do have on Windows is autohotkey. This is a magical piece of software. I am very annoyed that I hadn’t heard of it before 2024, because it’s basically a way to make Windows programmable and useful. It’s an entire programming environment where you can create GUIs, mimic user typing/clicking, and open programs.

I have barely scratched the surface of what you can do with autohotkey. To use it, you create script files with the ahk extension, run them with the autohotkey program, then it will hijack certain hotkeys to do whatever you want. If you mess up, and want the old hotkeys back, you can just click into “show hidden icons” in the bottom right of your Windows dock and exit the script.

There are lots of ways to set up snippets with autohotkey. For example, you can program it so every time you time “hmf” it will get converted to “Hello, my friend. I hope this email finds you well.” with the following script:

    ; autohotkey v2
    
    ::hmf::Hello, my friend. I hope this email finds you well.

I’ve started with a comment at the top specifying that it’s the V2 version of the scripting language. This is in because autohotkey recently went through a huge overhaul (a little like Python’s 3.0 change). Unfortunately, this makes for a lot of churn; lots of old scripts that no longer work. However, it has drastically improved the language; see Somehow AutoHotKey is kinda good now. It’s a little annoying for me, as my work signed off the 1.1 version of autohotkey, but I’m using 2.0 at home.

Anyway, back to snippets. The code above isn’t how I prefer to create snippets. I don’t really want text that I might accidentally type suddenly be hijacked. Instead, I like to take a windows hotkey that I barely ever use and use it to open a prompt with the list of snippets I can choose from. For me, the best hotkey is shift + windows + p. By default this hotkey configures display settings for multiple monitors, but I usually get to this by right clicking the desktop and opening display settings. I’m configuring monitors at the most once at the start of the day. Text snippets are more important, so I’m happy to lose the normal hotkey.

    ; autohotkey v2
    
    #SingleInstance force
    
    MyMenu := Menu()
    MyMenuBar := MenuBar()
    MyMenu := Menu.Call()
    MyMenuBar := MenuBar.Call()
    
    MyMenu.Add("1 For loop",PrintForLoop)
    MyMenu.Add("2 JS Function",PrintFunction)
    MyMenu.Add("3 Get users",PrintSelectUsers)
    
    PrintForLoop(ItemName, ItemPos, MyMenu){
            PrintSnipp("for (let i = 0; i < array.length; i++){`n`n}")
    }
    
    PrintFunction(ItemName, ItemPos, MyMenu){
            PrintSnipp("function myFunc(){`n`n}")
    }
    
    PrintSelectUsers(ItemName, ItemPos, MyMenu){
            PrintSnipp("SELECT TOP 10 * FROM tblUsers")
    }
    
    PrintSnipp(text){
            ClipSaved := ClipboardAll()
            A_Clipboard := ""
            A_Clipboard := text 
            ClipWait(2)
            send("^v")
            Sleep(100)
            A_Clipboard := ClipSaved
    }
    
    #+p::MyMenu.Show()

Just in case you have to use autohotkey 1.1 sometimes like me, here’s the script to do this in V1.1:

    ; autohotkey v1.1
    
    #SingleInstance force
    
    Menu SnippetMenu, Add, 1 For loop, PrintSnippet1
    Menu SnippetMenu, Add, 2 JS Function, PrintSnippet2
    Menu SnippetMenu, Add, 3 Get users, PrintSnippet3
    
    PrintSnippet1:
            PrintSnipp("for (let i = 0; i < array.length; i++){`n`n}")
    return
    
    PrintSnippet2:
            PrintSnipp("function myFunc(){`n`n}")
    return
    
    PrintSnippet3:
            PrintSnipp("SELECT TOP 10 * FROM tblUsers")
    return
    
    PrintSnipp(text){
            ClipSaved := ClipboardAll
            clipBoard := ""
            clipBoard = %text% 
            ClipWait,2
            if (!ErrorLevel) 
                    send, ^v
            Sleep, 100
            clipboard := ClipSaved
    }
    
    #+p::Menu, SnippetMenu, Show

After you run one of these scripts with your preferred version of autohotkey, it will mean shift + windows + p will open up a little menu where you can press 1, 2, or 3 and it will paste in the snippet you need.

I previously had some text files deep in my documents folder where I’d store important and useful text snippets that I’d always forget. Now, I just add them onto my autohotkey snippets, so I can pull them out immediately whenever I need.

From here you can go a lot of places. You could add submenus: This would mean you could split the scripts up by different languages or projects. You could also add a text input search to search through your snippets. Like I said, I’m scratching the surface. If you have any improvements or any better ways of getting snippets on your Windows machine, hit me up on Bluesky.

Scripting Audacity with JavaScript

2025-02-22T00:00:00Z

It's always great to get back to the programmer‘s favourite hobby: Spending more time automating something than it would take to just manually do the thing. I’ve been looking into audio recently and going into deep dives on the open source tools for removing noise from audio such as RNNoise: a machine learning model developed by Mozilla foundation for removing everything but voice from audio. When I looked at some of the open source packages which use RNNoise I couldn’t find one that was easy to use over the command line but I found a way to use a audacity plug-in to run RNNoise. I started using it and the changes it could make were pretty awesome for something open source.

Before I go on, I should also specify that this is going to be a mainly Windows post. I haven’t tested the scripts with Mac or Linux, but if you are ready and willing to add config for Linux or Mac, hit me up on bluesky!

Unfortunately audacity doesn’t have a command line interface. However, audacity does provide a way to interact with it using named pipes. This can be an insecure way to interact between processes, so they warn you not to use this on a webserver.

Before you’re able to use audacity’s scripting interface you’ll have to enable audacity’s mod-script-pipe module. to do this you need to click into edit then preferences then modules then make sure mod-script-pipe is enabled, and finally, restart audacity. Audacity provides two different pipes: one for sending commands to audacity and another one for receiving responses after commands have run. When the scripting module is enabled, then as soon as audacity starts, both pipes are opened and available to write to/read from. Windows named pipes are also quite frustrating in that after one is closed, you’re reliant on the server to restart the pipe, which Audacity doesn’t seem to reliably do.

This makes it slightly annoying to create a nice function that will run a command then wait for the response to be printed off. However, here’s some code that will let you do that. As you can see we’ve sent a command to audacity to list commands and audacity has returned all the commands:

    // Function to send commands to audacity:
    
    async function sendCommandToAudacity(command) {
        return new Promise((resolve, reject) => {
            // Open the command pipe for writing
            fs.open(commandPipePath, 'w', (err, commandFd) => {
                if (err) {
                    return reject(`Error opening command pipe: ${err}`);
                }
                // Write the command to the command pipe
                fs.write(commandFd, command, (err) => {
                    if (err) {
                        fs.close(commandFd, () => { }); // Close the file descriptor on error
                        return reject(`Error writing to command pipe: ${err}`);
                    }
                    // Close the command pipe after writing
                    fs.close(commandFd, async (err) => {
                        if (err) {
                            return reject(`Error closing command pipe: ${err}`);
                        }
                        const response = await readFromResponsePipe();
                        console.log(response)
                        resolve(response);
                    });
                });
            });
        });
    }
    function readFromResponsePipe() {
        return new Promise((resolve, reject) => {
            // Open the response pipe for reading
            let responseString = ''
            fs.open(responsePipePath, 'r', (err, responseFd) => {
                if (err) {
                    return reject(`Error opening response pipe: ${err}`);
                }
                // Read the response from the response pipe
                let currentResponseTime = 0;
                let lastTime = performance.now();
                function readMoreRecursive() {
                    if (responseString.trim() !== '' || currentResponseTime >= responsePipeTimeOut) {
                        // If the response we get back is empty, close the pipe and resolve:
                        fs.close(responseFd, (err) => {
                            if (err) {
                                return reject(`Error closing response pipe: ${err}`);
                            }
                            // Resolve with the response data
                            resolve(responseString);
                        });
                        return;
                    }
                    const buffer = Buffer.alloc(4096);
                    fs.read(responseFd, buffer, 0, buffer.length, null, (err, bytesRead) => {
                        if (err) {
                            fs.close(responseFd, () => { });
                            return reject(`Error reading from response pipe: ${err}`);
                        }
                        responseString = buffer.toString('utf8', 0, bytesRead);
                        readMoreRecursive()
                        currentResponseTime = currentResponseTime + (performance.now() - lastTime);
                        lastTime = performance.now();
                    });
                }
                readMoreRecursive();
            })
        })
    }

Great. How do we find more commands? To get a list of commands and the syntax to send them, you can find them in audacity macros. In Audacity, you can click into “Tools” then “Macro Manager” to look at macros. You can then create a new Macro and hit “New” then “Insert”.

From here you’ll have a list of different commands that audacity will expose. Many of them are options in the menus in audacity, along with a few other commands. After you’ve selected a few commands, you can save the macro, then export it. It will get exported as a text file which will show you the format you can use to send commands through the scripting pipe.

Macros are another way to automate editing audio files with audacity. Without the pipe module, you have to be in the audacity program to use macros, but this could make sense for your workflow: There are commands which will work for a generic use cases eg. noise reduction and truncating silence, but there will usually be custom edits you’ll have to make to specific parts of the file, so it can make sense to stay in the app.

Looking back at writing the JS scripts: I found that if you leave audacity open and rerun commands again, you’ll get errors because either audacity or node JS might not properly close the pipes (this also could be a skill issue on my part). Because of this, I wrote the script in a way that it will open an audio file, apply the commands, then exit audacity.

I’ve wrapped up the script into a class you can import and be able to use to start up audacity here - you can also find it on GitHub.

    // AudacityConnector.mjs
    import { spawn, exec } from 'child_process';
    import * as fs from 'fs';
    // Paths to the named pipes
    const commandPipePath = '\\\\.\\pipe\\ToSrvPipe';
    const responsePipePath = '\\\\.\\pipe\\FromSrvPipe'
    const openAudacityTimeOut = 10_000;
    const audacityStartUpTime = 5_000;
    const leaveOpenAfterCommands = false;
    var child;
    const defaultOptions = {
        audacityLocation: 'C:\\Program Files\\Audacity\\Audacity.exe',
        commandTimeOut: 30_000
    }
    export class AudacityConnector {
        constructor(options) {
            this.options = { ...defaultOptions, ...options };
        }
        openAudacity() {
            return new Promise((resolve, reject) => {
                child = spawn(this.options.audacityLocation, [], {
                    detached: false,
                    stdio: ['ignore', 'ignore', 'ignore']
                });
                let currentResponseTime = 0;
                let lastTime = performance.now();
                let isFound = false
                function pollForAudacity() {
                    setTimeout(() => {
                        if (isFound) {
                            return;
                        }
                        if (currentResponseTime > openAudacityTimeOut) {
                            console.log('Audacity did not open')
                            reject('Audacity did not open')
                        }
    
                        exec('tasklist', (err, stdout, stderr) => {
                            if (err) {
                                return reject(`Error executing tasklist: ${err}`);
                            }
                            if (stderr) {
                                return reject(`Error: ${stderr}`);
                            }
                            // Check if the output contains "audacity.exe"
                            if (stdout.toLowerCase().includes('audacity.exe')) {
                                isFound = true;
                                setTimeout(() => {
                                    resolve()
                                }, audacityStartUpTime)
                            }
                            currentResponseTime = currentResponseTime + (performance.now() - lastTime);
                            lastTime = performance.now();
                        })
                    }, 500)
                }
                pollForAudacity();
            })
        }
        closeAudacity() {
            child.kill()
        }
        sendCommandToAudacity(command) {
            return new Promise((resolve, reject) => {
                // Open the command pipe for writing
                fs.open(commandPipePath, 'w', (err, commandFd) => {
                    if (err) {
                        return reject(`Error opening command pipe: ${err}`);
                    }
                    // Write the command to the command pipe
                    fs.write(commandFd, command, (err) => {
                        if (err) {
                            fs.close(commandFd, () => { }); // Close the file descriptor on error
                            return reject(`Error writing to command pipe: ${err}`);
                        }
                        // Close the command pipe after writing
                        fs.close(commandFd, async (err) => {
                            if (err) {
                                return reject(`Error closing command pipe: ${err}`);
                            }
                            const response = await this.readFromResponsePipe();
                            console.log(response)
                            resolve(response);
                        });
                    });
                });
            });
        }
        readFromResponsePipe() {
            return new Promise((resolve, reject) => {
                // Open the response pipe for reading
                let responseString = ''
                fs.open(responsePipePath, 'r', (err, responseFd) => {
                    if (err) {
                        return reject(`Error opening response pipe: ${err}`);
                    }
                    function readMoreRecursive() {
                        if (responseString.trim() !== '') {
                            // If the response we get back is empty, close the pipe and resolve:
                            fs.close(responseFd, (err) => {
                                if (err) {
                                    return reject(`Error closing response pipe: ${err}`);
                                }
                                // Resolve with the response data
                                resolve(responseString);
                            });
                            return;
                        }
                        const buffer = Buffer.alloc(4096);
                        fs.read(responseFd, buffer, 0, buffer.length, null, (err, bytesRead) => {
                            if (err) {
                                fs.close(responseFd, () => { });
                                return reject(`Error reading from response pipe: ${err}`);
                            }
                            responseString = buffer.toString('utf8', 0, bytesRead);
                            readMoreRecursive()
                        });
                    }
                    setTimeout(() => {
                        fs.close(responseFd, () => { });
                        resolve()
                    }, this.options.commandTimeOut)
                    readMoreRecursive();
                })
            })
        }
    }

Then you can consume the audacity connector in your code like this (this script takes in a command line argument ):

    //script.js 
    
    const connector = new AudacityConnector();
    await connector.openAudacity();
    await connector.sendCommandToAudacity(`Import2: FileName=${import.meta.dirname}\\${process.argv[2]}`)
    await connector.sendCommandToAudacity("Select: Track=0")
    await connector.sendCommandToAudacity("TruncateSilence: Action=\"Truncate Detected Silence\" Compress=\"50\" Independent=\"0\" Minimum=\"1.0\" Threshold=\"-40\" Truncate=\"0.25\"")
    await connector.sendCommandToAudacity("ClickRemoval:Threshold=\"200\" Width=\"20\"")
    await connector.sendCommandToAudacity("LoudnessNormalization:DualMono=\"1\" LUFSLevel=\"-20\" NormalizeTo=\"0\" RMSLevel=\"-20\" StereoIndependent=\"0\"")
    await connector.sendCommandToAudacity(`Export2:Filename="${import.meta.dirname}\\${process.argv[3]}" NumChannels="1"`)
    connector.closeAudacity();

Then run it with:

    node ./script.js myaudiofile.wav out.wav

If you want to try the RNNoise filter, then you can use the werman plugin for audacity. The results you can get are pretty amazing. You can make an iPhone recording sound like it was from a professional studio. To use it with the above script:

    await connector.sendCommandToAudacity("RnnoiseSuppressionForVoice: Use_Preset=\"Default\"")

Here are other open source command line tools for audio processing you can check out if you want other ways to process your audio:

A tale of two static site hosts

2025-11-01T00:00:00Z

This site has been moved to Cloudflare pages. It used to be hosted on Azure blob storage behind Azure CDN. So what happened? Why move?

One day I got an email from Azure "Hey, you need to migrate away from Azure CDN classic, and move to either Azure Front Door standard or premium". Okay fine, this is a personal site that gets very little traffic, so I'll use Front Door standard I guess. Not the premium one. I upgraded to it. Usually my hosting costs have been about $2.00 per month. After migrating to Front Door: a $22.00 bill . Okay fine, I'll remove Azure Front Door and just point my DNS straight to the static site.

Uh oh - now I've got certificate errors. HTTPS is broken. The site is insecure. I didn't have time to fix this for about a month, so my site was only available after clicking through a bunch of security notices. I've always struggled with the user experience of Azure. Using blob storage to host a static site has always felt awkward, even though it's a common use case. So, finally I decided enough is enough and to migrate the site to Cloudflare pages.

What was the experience like moving to Cloudflare pages? I went to Cloudflare and:

Registered using Google single sign-on
Clicked "new application", "Pages"
I authenticated to my GitHub and pointed Cloudflare to the repo
Chose the 11ty option for the type of site.

Then the build started. My developer brain kicked in: "How is this going to fail? What configuration changes am I going to need to make? What version of 11ty and node.js are cloudflare running? Are they going to point to the correct directory for the site root?". Then the log said "build successful". They gave me a URL and my site was live at a Cloudflare URL. I updated my custom domain in Cloudflare and my domain host. Then I was finished.

And the site is FAST. Like, crazy fast. At no point in this process had I given them a credit card or anything. Cloudflare pages are basically free until you become famous. It's almost annoying how good this experience is.

Have other people come across the same issue with Azure blob storage static sites? YES! Eakan Gopalakrishnan has basically written the same blogpost as this. Also on reddit, there are a bunch of posts from people with static sites on Azure mad about these changes. This seems to be a pretty common problem for a lot of people hosting their sites! In Microsoft's defence, people using blob storage to host their static sites for $2.00 a month are not customers that get Azure profitable. We probably cost more in administration costs than we give to Azure. We aren't worth worrying about from a money-making perspective. Azure make their money from big enterprises burning millions on Cloud services. I don't matter in the scheme of things.

But how much goodwill can you burn? A lot of the people hosting these static sites will be developers like myself. If we get the rug pulled under us by a cloud provider in our fun personal sites, how are we going to trust the same provider at work where there's real money on the line?

It reminds me a little bit of Facebook. In 2009, we used Facebook to connect and see posts from friends, and I loved writing posts. Now after years of repeated rug-pulls and breaches of trust, I don't want to invest anything into Facebook. It's 45% slop, 45% ads and 10% things related to my friends and family. I'm only there because I have to be. The cultural relevance of Facebook is not what it once was. I certainly wouldn't develop any apps or games on top of Facebook's platform because I couldn't trust them not to change the rules underneath me.

Now Azure have burned a bunch of goodwill on these CDN changes, and Cloudflare have earned my goodwill by making such a smooth onboarding process and giving away low-traffic sites for free. The question is, does this goodwill actually matter? To the immediate bottom line of these companies, probably not. However, I think there's only so many rug pulls you can do before you become the Facebook of Cloud platforms. Before you lose your relevance.

Now here come the caveats. Who remembers Kiwifarms? It was (is?) a site to doxx trans people and make their lives hell. Any respectable company should have refused that website service and told it to take a hike. Cloudflare were proudly defending it against DDOS attacks. They made the wrong call on that.

On a more selfish note: it has been good for me career-wise to practice a little bit of Azure configuration. It's not something that I enjoy, but it's something that I have had to do for work. It's good to eat your vegetables. A little practice using Azure for my personal site made me slightly less lost configuring Azure websites and databases at work.

Stay safe out there, and don't get rug-pulled.

Hugh Haworth - Developer Blog

2025-11-01T09:28:48Z

Hugh Haworth - Developer Blog

Writing about the cool things we do with connected computers

A tale of two static site hosts

1 November 2025

Azure to Cloudflare pages

Scripting Audacity with JavaScript

22 February 2025

More ear happiness with more automation

Snippets everywhere on Windows with autohotkey

31 December 2024

Text at your fingertips

Speed up Power BI merges with record lookups

14 November 2024

Lookup lightning

OIDC Authentication in ASP.NET Core apps explained

14 October 2024

Leave the login forms to the professionals

OIDC Authentication in server-side Blazor apps

17 July 2024

Microsoft, can you tell me who this is? (again)

What's in a PDF

11 March 2024

Come with me ... and you'll be ... in a world of pure documentation

Ways of getting data from Excel files into a Database

18 January 2024

Gettin' data where it needs to go

Sticker Charts

13 August 2023

My 3 Gripes with Power BI

DotNetCore, Azure AD, and SAML

20 April 2023

Microsoft, can you tell me who this is?

Hill Times, Vim and Delayed Gratificication

31 January 2023

Running up that hill

Goodbye Heroku free plan

8 January 2023

Death of a Salesforce

Gizmo Girls

1 December 2022

A new blog about an old TV show

Hosting small websites on EC2

2 October 2022

Embrace the Bezos

Printing a file with the Apache Portable Runtime

31 July 2022

Getting portable with C

Compile your own Apache Module in C

16 June 2022

Extending servers

Why is SQL weird?

10 May 2022

Fear and loathing in RDBMS

The callbacks are out to get you

3 March 2022

The part of JavaScript that beginners struggle with

My JavaFX ListViews are doubling up

18 January 2022

Getting lists in JavaFX to behave

Explorations in C, C++, Win32, and SDL

10 January 2022

Allocatin' and commemoratin'

Intro to RequestAnimationFrame

22 December 2021

An example-first guide to getting things moving

What should “create” do?

29 September 2021

Creating content in a cloud-first world

React and Electron

2 September 2021

Of chromium instances and create-react-apps

Rollup Plugins for Absolute Beginners

26 July 2021

Start getting your hands dirty in the build step

Reviving an old laptop with linux

21 July 2021

Bringing an old machine back from death

HTTP Caching is a Superpower

11 May 2021

Make the web faster the old way

Going static

25 January 2021

Migrating from PHP land to Eleventy

Easy Node Live Server

17 January 2021

Tired of hitting Ctrl R then Ctrl F5 all the time?

Dropping Plantr 2.0

12 January 2021

Get gardening like it's 2021

2020 to 2021

31 December 2020

Learning web development in the year from hell and plans for 2021

Tailwind for interactions - Alpine.JS

7 December 2020

Who put JavaScript in my HTML?!

React on a Budget with Lit-html and Haunted

17 November 2020

Into a build-step free future

Batteries-included React

16 October 2020

The myriad of options we have to supercharge Jordan Walke's framework.

Eloquent Models - First Impressions

5 October 2020

My First impressions with Laravel's solution to data management

Upcoming project Plantr 2.0

11 September 2020

Some things I learnt building a gardening app and how I'm going to make a new one.

Web Colour Deep Dive

6 September 2020

A look into the world of RGB, hue, saturation, lightness, hex codes and colours on the web and ways to process them.

PHP Data Objects

27 August 2020

A cowboy's guide to prepared PDO Statements

Building this Blog

20 August 2020

The joys of reinventing the wheel - building this blog with Edoras.

Home