1652820308 The Code Is the To-Do List https://www.executeprogram.com/blog/the-code-is-the-to-do-list

Managing our focus during development can be a challenge.\nHere's a common scenario: we're in the middle of working on user.ts when we find ourselves thinking: "I really need to go change this other thing in login.ts".\nBut it's rarely a good idea to interrupt ourselves during a complex change.\nWe have a lot of context in our heads that we don't want to lose.

There are two common ways to deal with this: we can write down a note that reminds us to do it later, or we can write a quick failing (or pending) test that forces us to do it later.

Both approaches work just fine.\nHowever, there's a third way that's superior to both in some situations: we can use the code itself as a to-do list.\nSpecifically, we can use the linter to enforce our to-dos.

In Execute Program, we use ESLint with the "no-warning-comments" lint rule enabled:

module.exports = {\n  "rules": {\n    "no-warning-comments": ["error", {terms: ["xxx"], location: "anywhere"}],\n    ...\n  },\n  ...\n}\n

This configuration makes any comment containing "XXX" an error.\nLint errors cause our CI builds to fail, which means our XXX comments block deployment to production.\nAs a result, we can now use XXX comments to mark critical problems that should never reach production.

(We could also use TODO comments instead of XXX by changing the ESLint configuration above.\nBut our convention is that TODO comments are for longer-term deferred maintenance like "remove this old API endpoint" and XXX means "do not let this go to production".\nThe particular string "XXX" isn't important here.)

This approach is often better than separate to-do lists and failing tests because:

Blocking deployment makes XXX comments more reliable than separate notes.\nWe might accidentally deploy code without addressing notes in a separate system, but the ESLint rule above prevents deploys that contain "XXX" comments.
Not all changes are easily expressed as tests.
Leaving a comment in the code is usually faster than writing a test or switching to a note-taking app.
Everyone working in the code can see the XXX comments, with no need to agree on a shared note-taking system.

In the rest of this post, we'll see how XXX comments have helped our development process.\nWe'll review four real examples from the Execute Program codebase, ordered from simplest to most complex.\nThese examples come from a feature branch that's diverged from our main branch by about 3,500 lines.

$ git diff main --stat | tail -1\n 40 files changed, 2448 insertions(+), 1155 deletions(-)\n

There are a lot of moving parts in this release, so it's crucial to keep track of all the changes we need to make as we develop!

1. Simplifying code

// XXX: Remove this module; it's trivial now.\n

We have a module that used to export several TypeScript types.\nOn this branch, all but one of those types are now gone.\nWe don't need this whole module to export one small type.\nThe type can live in the subsystem's top-level module instead.

This change feels quick, so it's tempting to make it immediately.\nBut it will be just as fast if we do it tomorrow.\nWe don't have any special context in our heads that makes this change faster today.

On the other hand, we probably have a lot of context in our heads about the bigger change that we were making when we noticed this.\nProgramming is highly dependent on short-term memory, and any distraction is a chance for us to forget details that may be important.\nLeaving an XXX comment only takes a few seconds, keeps our attention on the current task, and ensures that we'll remember to remove this module later.

2. Deferring a data format change

// XXX: This is inserting "Start Case" property names. They should be "camelCase".\n

While working on this branch, we needed to change a data format.\nUnlike the type definition change above, updating the rest of the code to match the new format will take a substantial amount of work.\nThis change will also impact some other parts of the system, so we'd rather isolate it in a separate commit.

Like the module removal example, we don't want to distract ourselves.\nAnd the need for a separate commit makes deferring this change an even easier decision.

Sometimes, the fine-grained version control details also matter.\nYou might think: Why not make the change now, then tease the two changes apart after the fact with git add -p or an interactive rebase?\nIn this case, many of the affected lines are very close to lines that we've already changed.\nIndividual modifications from both of our changes would end up in the same diff hunks, which makes teasing them apart much more difficult.\nThe version control wrangling could end up taking longer than the change itself.

3. Managing temporary environment changes

// XXX: Changed the number here for faster testing. Change it back.\n

Our code has an event-triggering threshold: when event A happens enough times, it triggers event B.\nOn this branch, we sometimes want to manually trigger event B after only a few occurrences of A, so we've temporarily lowered this threshold.\nBut we don't want to accidentally push that change out, so we added an XXX comment.

Because we're working on a very stable part of the system, it doesn't make sense to lower the threshold permanently for our development environment.\nChanging the threshold based on the environment adds application complexity.\nWorse, it's yet another way that dev diverges from production, which is never desirable.\nWe decided that it was better for this branch to diverge temporarily than for the dev environment to diverge permanently.

An XXX comment allows us to keep using the lower threshold, and gives us peace of mind that we won't accidentally push this dangerous change to production.

4. Documenting our work

The application-specific details are removed from this example.\nBut its structure is as we wrote it in the real code.

/* XXX: Write summary comment explaining this module\n *   - Mention that the foo() function is responsible for frobbing.\n *   - Mention that bar() hits Stripe, so avoid multiple calls to it. But none\n *     of the other functions is allowed to hit Stripe, so call them at will.\n */\n

This branch introduces a major new module to the system.\nThe module will need a summary comment, so it's tempting to write it right away.\nBut writing module-level comments early on can be risky.

First, we may end up reverting our current changes, and this module may not even exist when the branch lands.\nIf we write a perfect, nicely-formed comment, that effort might be wasted.

Second, this module or the other modules around it might change during the lifetime of our branch.\nWe'll have a better sense for how the module fits into the system once the branch is stabilized, so that's a better time to finalize a high-level comment.

Finally, even if neither of those apply, the usual reasoning does still apply: we don't want to interrupt the change that we're making right now.

Even if we don't write the big comment up front, we can still make notes about what to include in it.\nUsing a bulleted list lets us capture notes in the moment, without needing to take the time to write a full, nicely-formatted comment.\nThis is a great example of how using the code as a to-do list balances tracking future work vs. interrupting our current task.

ESLint in action

Once the comments are in place, we can see them in our ESLint output.\nOur dev script automatically runs ESLint on every source file change, so we'll see this continually.\nThis is also the output that will show up in CI, blocking deploys to production.

(The filenames are anonymized here because the exact details of the new subsystem aren't important.)

[eslint:server] src/server/old-subsystem/old-file.ts\n[eslint:server]   63:13  warning  Unexpected 'xxx' comment: 'XXX: This is inserting "Start Case"...'  no-warning-comments\n[eslint:server]\n[eslint:server] src/server/new-subsystem/types.d.ts\n[eslint:server]   5:1  warning  Unexpected 'xxx' comment: 'XXX: Remove this module; it's trivial...'  no-warning-comments\n[eslint:server]\n[eslint:server] src/server/new-subsystem/index.ts\n[eslint:server]    7:1   warning  Unexpected 'xxx' comment: 'XXX: Write summary comment explaining...'    no-warning-comments\n[eslint:server]   94:11  warning  Unexpected 'xxx' comment: 'XXX: Changed the number here for faster...'  no-warning-comments\n[eslint:server]\n[eslint:server] ✖ 4 problems (0 errors, 4 warnings)\n[eslint:server]\n[eslint:server] ESLint found too many warnings (maximum: 0).\n

Development flow

In large changes, it's common for us to have dozens of these XXX lint errors live at once.\nUsually, the arc of a large change is:

Begin with the most risky parts of the change.\nIt's usually best to start with the riskiest part.\nIf the risks turn into serious problems, we may reprioritize or even cancel this project.\nWe want that to happen at the beginning, rather than after we've spent a lot of effort on it.
As we do the most risky work, we defer other work that we encounter along the way, adding XXX comments.
At some point, we reach "peak XXX", which is often dozens of XXX comments at once.
Then we begin resolving them until there are none left.

By the time we're resolving XXX comments, some will be outdated due to other changes made after we added the comment.\nThat's fine; it's just a comment, and we can delete it.\n(But if we'd made the change at the time, that work would've been wasted!)

Even when the comments are still current, they're often quick changes that were just a bit too big to distract ourselves with at the time.\nWe estimate that the downward slope from "peak XXX" to "0 XXX" is usually around 10% of the total development time for a branch.\nAll of that work was going to happen one way or another; we just shifted it in time to aid our concentration and to allow better risk prioritization.

Using the linter to block XXXs from going to production is a great alternative to keeping a separate to-do list, or writing a test for every change we think of.\nWriting XXX comments isn't just faster; it protects our focus and prioritizes our attention.\nGive this approach a try on your next project!

\n html 1642542308 TypeScript Features to Avoid https://www.executeprogram.com/blog/typescript-features-to-avoid

This post lists four TypeScript features that we recommend you avoid.\nDepending on your circumstances, there may be good reasons to use them, but we think that avoiding them is a good default.

TypeScript is a complex language that has evolved significantly over time.\nEarly in its development, the team added features that were incompatible with JavaScript.\nRecent development is more conservative, maintaining stricter compatibility with JavaScript features.

As with any mature language, we have to make difficult decisions about which TypeScript features to use and which to avoid.\nWe experienced these trade-offs firsthand while building Execute Program's backend and frontend in TypeScript, and while creating our comprehensive TypeScript courses.\nBased on our experience, here are four recommendations about which features to avoid.

I. Avoid enums (See our lesson)

Enums give names to a set of constants.\nIn the example below, HttpMethod.Get is a name for the string 'GET'.\nThe HttpMethod type is conceptually similar to a union type between literal types, like 'GET' | 'POST'.

enum HttpMethod {\n  Get = 'GET',\n  Post = 'POST',\n}\nconst method: HttpMethod = HttpMethod.Post;\nmethod; // Evaluates to 'POST'\n

Here's the argument in favor of enums:

Suppose that we eventually need to replace the string 'POST' above with 'post'.\nWe change the enum's value to 'post' and we're done!\nOther code in the system only references the enum member via HttpMethod.Post, and that enum member still exists.

Now imagine the same change with a union type instead of an enum.\nWe define the union 'GET' | 'POST', then later we decide to change it to 'get' | 'post'.\nAny code that tries to use 'GET' or 'POST' as an HttpMethod is now a type error.\nWe have to update all of that code manually, which is an extra step when compared to enums.

This code maintenance argument for enums isn't very strong.\nWhen we add a new member to an enum or union, it rarely changes after creation.\nIf we use unions, it's true that we may have to spend some time updating them in multiple places, but it's not a big problem because it happens rarely.\nEven when it does happen, the type errors can show us which updates to make.

The downside to enums comes from how they fit into the TypeScript language.\nTypeScript is supposed to be JavaScript, but with static type features added.\nIf we remove all of the types from TypeScript code, what's left should be valid JavaScript code.\nThe formal word used in the TypeScript documentation is "type-level extension": most TypeScript features are type-level extensions to JavaScript, and they don't affect the code's runtime behavior.

Here's a concrete example of type-level extension.\nWe write this TypeScript code:

function add(x: number, y: number): number {\n  return x + y;\n}\nadd(1, 2); // Evaluates to 3\n

The compiler checks the code's types.\nThen it needs to generate JavaScript code.\nFortunately, that step is easy: the compiler simply removes all of the type annotations.\nIn this case, that means removing the : numbers.\nWhat's left is perfectly legal JavaScript code.

function add(x, y) {\n  return x + y;\n}\nadd(1, 2); // Evaluates to 3\n

Most TypeScript features work in this way, following the type-level extension rule.\nTo get JavaScript code, the compiler simply removes the type annotations.

Unfortunately, enums break this rule.\nHttpMethod and HttpMethod.Post were parts of a type, so they should be removed when TypeScript generates JavaScript code.\nHowever, if the compiler simply removes the enum types from our code examples above, we're still left with JavaScript code that references HttpMethod.Post.\nThat will error during execution: we can't reference HttpMethod.Post if the compiler deleted it!

/* This is compiled JavaScript code referencing a TypeScript enum. But if the\n * TypeScript compiler simply removes the enum, then there's nothing to\n * reference!\n *\n * This code fails at runtime:\n *   Uncaught ReferenceError: HttpMethod is not defined */\nconst method = HttpMethod.Post;\n

TypeScript's solution in this case is to break its own rule.\nWhen compiling an enum, the compiler emits extra JavaScript code that never existed in the original TypeScript code.\nFew TypeScript features work like this, and each adds a confusing complication to the otherwise simple TypeScript compiler model.\nFor these reasons, we recommend avoiding enums and using unions instead.

Why does the type-level extension rule matter?

Let's consider how the rule interacts with the ecosystem of JavaScript and TypeScript tools.\nTypeScript projects are inherently JavaScript projects, so they often use JavaScript build tools like Babel and webpack.\nThese tools were designed for JavaScript, and it's still their primary focus today.\nEach tool is also an ecosystem of its own.\nThere's a seemingly-endless universe of Babel and webpack plugins to process code.

How can Babel, webpack, their many plugins, and all of the other tools and plugins in the ecosystem fully support TypeScript?\nFor most of the TypeScript language, the type-level extension rule makes these tools' jobs relatively easy.\nThe tools strip out the type annotations, leaving valid JavaScript.

When it comes to enums (and namespaces, which we'll see in a moment) things are more difficult.\nIt's not good enough to simply remove enums.\nThe tools have to turn enum HttpMethod { ... } into working JavaScript code, even though JavaScript doesn't have enums at all.

This brings us to the practical problem with TypeScript's violations of its own type-level extension rule.\nTools like Babel, webpack, and their plugins are all designed for JavaScript first, so TypeScript support is just one of their many features.\nSometimes, TypeScript support doesn't receive as much attention as JavaScript support, which can lead to bugs.

The vast majority of tools will do a good job with variable declarations, function definitions, etc.; all of those are relatively easy to work with.\nBut sometimes mistakes creep in with enums and namespaces, because they require more than just stripping off the type annotations.\nYou can trust the TypeScript compiler itself to compile those features correctly, but some rarely-used tools in the ecosystem may make mistakes.

When your compiler, bundler, minifier, linter, code formatter, etc. silently miscompiles or misinterprets an out-of-the-way piece of your system, it can be very difficult to debug.\nCompiler bugs are notoriously difficult to track down.\nNote these words: "over the week, with the help of my colleagues, we managed to get a better understanding of the scope of the bug."\n(Emphasis added.)

II. Avoid namespaces (See our lesson)

Namespaces are like modules, except that more than one namespace can live in a single file.\nFor example, we might have a file that defines separate namespaces for its exported code and for its tests.\n(We don't recommend doing this, but it's a simple way to show off namespaces.)

namespace Util {\n  export function wordCount(s: string) {\n    return s.split(/\\b\\w+\\b/g).length - 1;\n  }\n}\n\nnamespace Tests {\n  export function testWordCount() {\n    if (Util.wordCount('hello there') !== 2) {\n      throw new Error("Expected word count for 'hello there' to be 2");\n    }\n  }\n}\n\nTests.testWordCount();\n

Namespaces can cause problems in practice.\nIn the section on enums above, we saw TypeScript's "type-level extension" rule.\nNormally, the compiler removes all of the type annotations, and what's left is valid JavaScript code.

Namespaces break the type-level extension rule in the same way as enums.\nIn namespace Util { export function wordCount ... }, we can't remove the type definitions.\nThe entire namespace is a TypeScript-specific type definition!\nWhat would happen to the other code outside of the namespace calling Util.wordCount(...)?\nIf we delete the Util namespace before generating JavaScript code, then Util doesn't exist any more, so the Util.wordCount(...) function call can't possibly work.

As with enums, the TypeScript compiler can't simply delete the namespace definitions.\nInstead, it has to generate new JavaScript code that doesn't exist in the original TypeScript code.

For enums, our suggestion was to use unions instead.\nFor namespaces, we recommend using regular modules.\nIt may be a bit annoying to create many small files, but modules have the same fundamental functionality as namespaces without the potential downsides.

III. Avoid decorators (for now)

Decorators are functions that modify or replace other functions (or classes). Here's a decorator example taken from the official docs.

// This is the decorator.\n@sealed\nclass BugReport {\n  type = "report";\n  title: string;\n\n  constructor(t: string) {\n    this.title = t;\n  }\n}\n

The @sealed decorator above alludes to the C# sealed modifier, which prevents other classes from inheriting from the sealed class.\nWe'd implement it by writing a sealed function that takes a class and modifies it to prevent inheritance.

Decorators were added to TypeScript first, before beginning their standardization process in JavaScript (ECMAScript).\nAs of January 2022, decorators are still a stage 2 ECMAScript proposal.\nStage 2 is "draft".\nThe decorator proposal also seems to be stuck in committee purgatory: it's been at stage 2 since February of 2019.

We recommend avoiding decorators until they're at least a stage 3 ("candidate") proposal, or stage 4 ("finished") for more conservative teams.

There's always a chance that ECMAScript decorators will never be finalized.\nIf that happens, they'll end up in a similar situation to TypeScript enums and namespaces.\nThey'll continue to break TypeScript's type-level extension rule forever, and they'll be more likely to break when using build tools other than the official TypeScript compiler.\nWe don't know whether that will happen or not, but the benefits of decorators are minor enough that we'd rather wait and see.

Some open source libraries, most notably TypeORM, use decorators heavily.\nWe recognize that following our recommendation here precludes using TypeORM.\nUsing TypeORM and its decorators is a fine choice, but it should be done intentionally, recognizing that decorators are currently in standardization purgatory and may never be finalized.

IV. Avoid the private keyword (See our lesson)

TypeScript has two ways to make class fields private.\nThere's the old private keyword, which is specific to TypeScript.\nThen there's the new #somePrivateField syntax, which is taken from JavaScript.\nHere's an example showing each of them:

class MyClass {\n  private field1: string;\n  #field2: string;\n  ...\n}\n

We recommend the new #somePrivateField syntax for a straightforward reason: these two features are roughly equivalent.\nWe'd like to maintain feature parity with JavaScript unless there's a compelling reason not to.

To recap our four recommendations:

Avoid enums.
Avoid namespaces.
Favor #somePrivateField over private somePrivateField.
Hold off on using decorators until they're standardized.\nIf you really need a library that requires them, consider their standardization status when making that decision.

Even when avoiding these features, it's good to have a working knowledge of them.\nThey show up often in legacy code, and even in some new code.\nNot everyone agrees that they should be avoided.\nExecute Program's TypeScript courses teach these features for the reasons that we explained in our post on Teaching the Unfortunate Parts.

\n html 1604958308 Teaching the Unfortunate Parts https://www.executeprogram.com/blog/teaching-the-unfortunate-parts

When explaining a technology, we have to decide how to approach its shortcomings.\nThere might be mistakes in its design, or it might have usability problems, or it might be unreliable.\nHow do we approach these and how much emphasis do we place on them?

One approach is: "This tool has a lot of problems, but we'll show you how to avoid them."\nThat can demotivate the learner: "Why am I learning this thing if it has so many problems?"

Another common approach is to never call out the problems at all.\nThat can also demotivate the learner, but the demotivation is delayed.\nWhen they finish your video, book, etc. and use the tool in practice, they suddenly encounter the sharp edges.\nNow they might think "I haven't heard about these problems so this must be my fault."

In Execute Program, we try to avoid both of the above approaches.\nInstead, we aim for neutral descriptions in our courses.\nWhere problems exist, we describe them flatly and directly.\nOften, probably due to personal writing quirks, our descriptions involve the word "unfortunate".

Here are four times when we used the word "unfortunate" or "unfortunately" in a lesson.\nEach is expanded here because the original surrounding lesson context is missing, but the original "unfortunate" is retained in each example.

1. NaN

(From the Modern JavaScript / isNaN lesson.\nThe lesson links in this post, including that one, will take you directly to the interactive lesson, which doesn't require registering an account.)

Any operation on a NaN returns another NaN.\nUnfortunately, that means that NaNs will propagate through the system.\nBy the time you actually see the NaN, it might have ended up in a very different place from where it started.

> const lastIndex = ['a', 'b', 'c'].elngth - 1;\n  const middleIndex = Math.floor(lastIndex / 2);\n  middleIndex;\nNaN\n

Note the typo: elngth.\nThat returns undefined, then we subtract 1 from it giving NaN, then we divide by 2, then we call Math.floor.\nNow imagine that each of those steps happened in a different function.

All we know is that a NaN came out the end.\nNow we have to ask the perpetual question in dynamic languages: which function (or class, or module) caused the NaN (or null, undefined, nil, None, etc.), and which functions merely propagated it?

NaN itself is part of the IEEE 754 floating point standard.\nIt behaves the way it does for a reason.\nBut this is still a JavaScript problem because JavaScript returns NaN in so many cases where other languages don't.\nIn our example above, the NaN showed up when we did undefined - 1.\nOther dynamic languages (like Ruby or Python) throw an error when we try to do that, and most static languages prohibit it at compile time.\nThis is a problem specific to JavaScript.

2. Mapping over maps

(From the Modern JavaScript / Maps lesson.)

Modern versions of JavaScript support a map data type.\nIt's kind of like an object where the keys can be any type, not just strings.\nJavaScript also has a map method on arrays, which transforms the array's values into other values.

The data type Map and the method map are related at the conceptual level: they both "map" (or "relate") things to other things.\nOther than that, they have no relationship.\nTheir identical names are just an unfortunate accident of history.

3. No negative array indexes in JavaScript

(From the JavaScript Arrays / Negative array indexes lesson.)

Ruby and Python both allow negative indexing on arrays.\nIn those languages, arr[-1] means "the last element of the array".\nJavaScript has a concept of negative indexes: we can call arr.slice(-1) to get an array containing only the last element of arr.

We might expect that to work for normal array access as well, as it does in Ruby and Python: someArray[-1].\nAlternately, we might expect someArray[-1] to throw an exception.\nUnfortunately, it doesn't do either of those.\nInstead, it returns undefined, the same value that we get when accessing any other index that doesn't exist.

> const arr = ['a', 'b', 'c'];\n  arr[-1];\nundefined\n

4. TypeScript's unsoundness

(From the TypeScript / Type soundness lesson.)

A sound type system is one that fully enforces its own rules.\nTypeScript is unsound, which means that the compiler will sometimes accept code that violates the compiler's own rules.

In the example below, we create an array of strings.\n"Array of strings" means that it should never contain anything else.\nIt certainly shouldn't contain any numbers.

Then we assign the array to a new variable with the type string | number.\nThat lets us push a number into the array.\nWe've now violated the original array variable's type: it says it's a string[], but it contains both strings and numbers.

> let names: string[] = ['Amir', 'Betty'];\n  let unsoundNames: (string | number)[] = names;\n  \n  // This is unsound! It causes the "names" array to contain a number!\n  unsoundNames.push(5);\n  names;\n['Amir', 'Betty', 5]\n

This is very bad, and unfortunately there's no way to fix it.

Being honest about problems

It's important to acknowledge design problems, especially when teaching.\nLearners are new to the topic, so they rarely have enough context to analyze designs critically.\nFor example, someone learning TypeScript as their first static language won't know what soundness is.

People are going to hit TypeScript's unsoundness in real code.\nThey'll find NaNs propagating through their JavaScript systems running in production.\nThey'll wonder why the map data type and the map method have the same name even though they don't interact.\nTheir first thought will be that they're just missing something.

As authors, we could try to head this off with "OK, this technology has a ton of problems... in fact, it's pretty bad, but here we go, let's learn it!"\nThat sets the learner up for demotivation from the start.\nAlternately, we could describe only the good parts of the tool, staying silent about the sharp edges.\nBut that sets the learner up for demotivation later when they encounter sharp edges and blame themselves.

It's better to describe the good parts, then tell the learner that, unfortunately, TypeScript's type system is also unsound.\nBut it's unsound for a reason: it's a trade-off that its designers made to achieve compatibility with the existing JavaScript ecosystem.\nNow the learner knows that the problem is real and they're not imagining it; they know that the problem isn't their fault; and, if the lesson did its job, they know how to spot the problem and work around it.

\n html 1593117908 TypeScript's excess properties can bite you https://www.executeprogram.com/blog/typescripts-excess-properties-can-bite-you

Recently, our Cypress tests started hanging silently.\nThis post explains the cause: because of a subtle part of TypeScript's language design, some circular data structures were invisible in the code, which caused a serialization error, which Cypress handled incorrectly.\nIf we don't understand the subtleties of our tools, bugs like this can be inscrutable; but if we learn the subtleties, they can be quick, if annoying, speed bumps.

1. Excess object properties

TypeScript's type system is unusual in many ways.\nHere's an example highlighting one of them:

type User = {\n  name: string,\n}\n\nfunction buildAmir() {\n  return {\n    name: 'Amir',\n    age: 36,\n  }\n}\n\nconst amir: User = buildAmir()\n\nconsole.log(amir)\n

In most statically typed languages, this code wouldn't compile, but TypeScript allows it.\nWhat comes out when we run it?\nAmir has the type {name: string}, so it seems like the answer should be {name: 'Amir'}.\nWhat actually comes out at runtime is:

{name: 'Amir', age: 36}\n

This arises from two design decisions in TypeScript.\nFirst, objects are allowed to have excess keys and still satisfy an object type that doesn't have those keys.\nThe excess keys are ignored by the type system.\nFor example, the compiler wouldn't let us say amir.age, even though the age property does exist at runtime!

Second, the TypeScript compiler emits code by stripping the types away.\nWhat's left is JavaScript.\nFor our example, the compiler emits JavaScript roughly like this:

function buildAmir() {\n  return {\n    name: 'Amir',\n    age: 36,\n  }\n}\n\nconst amir = buildAmir()\n\nconsole.log(amir)\n

When we look at it that way... of course the output is { name: 'Amir', age: 36 }.\nThat's what the code says!

At first glance, this all seems fine.\nSo how did it make our tests silently fail to run?

2. Large objects

Internally, Execute Program has an Example type representing a single interactive code example in a lesson.\nIts type looks like this, heavily simplified and tweaked to make sense in isolation.

type Example = {\n  kind: "example"\n  id: string\n  code: string\n  expectedResult: string\n  isInteractive: boolean\n}\n

For example, we might have an example where code is '1 + 1' and expectedResult is '2'.\nIn that case, Execute Program would show the user 1 + 1 and wait for them to type 2.

In the past, the lessons referenced their examples, but the examples didn't know about the lessons that contain them.\n(There's no lesson property in the object above.)\nThat was done because backreferences would create a reference cycle: the lesson references the example, which references the lesson, which references the example, and so on forever.\nReference cycles make garbage collectors work harder, as well as causing confusing bugs like the one that we're working toward here.

(We'll use "reference cycle" and "circular data" interchangeably in this post.)

It's inconvenient for examples not to know about their lessons, so we recently added some lesson data to the example type.\nBut not a full lesson, just a little bit of data: the containing lesson's ID.\nThis made a particular new feature much easier to implement.\nNow the Example type becomes:

type Example = {\n  // This stuff is all the same as before.\n  kind: "example"\n  id: string\n  code: string\n  expectedResult: string\n  isInteractive: boolean\n\n  // This is new.\n  lesson: {\n    id: string\n  }\n}\n

No reference cycles!\nThat little lesson object only has an id, not a list of examples that would cause a cycle.

But unfortunately, we naively created the example object like this:

const example: Example = {\n  kind: "example",\n  id,\n  code,\n  expectedResult,\n  isInteractive,\n  lesson,\n}\n

The lesson there is a full lesson object.\nFrom TypeScript's perspective, example.lesson's type is {id: string}.\nBut at runtime it will be the entire lesson object, which is (1) huge, and (2) contains references to all of the lesson's examples, which contain references back to the lesson, which contains references to all of the examples, and so on.\nThe static types show no reference cycle, but one still exists.

3. Test data generation

Our Cypress tests need to load up our actual lessons for testing.\nUnfortunately, there's not a great way to do that in Cypress.\nWhat we do today is:

Use a custom Cypress plugin (they're easy to write, fortunately) that runs before the Cypress test browser is launched.
The plugin looks at the fully loaded curriculum object, which is loaded from our lesson files in the same way that it's loaded in the normal app.
The plugin dumps that curriculum object (including courses, which contain lessons, which contain examples) to a JSON file.
The tests can then import that JSON file to access the lesson data.

This seems clunky because it is clunky.\nBut Cypress comes with a lot of constraints, and this was how we satisfied them.

When we added lesson to the Example object, the clunks came home to roost.\nTrying to call JSON.stringify on any circular data structure will cause an exception in Node.\nHere's a shell command demonstrating:

$ node -e 'const x = {}; x.x = x; JSON.stringify(x)'\nTypeError: Converting circular structure to JSON\n

The bug

The JSON.stringify call on our curriculum object definitely threw an exception inside our Cypress plugin code.\nBut the way that manifested in Cypress was... nothing.\nCypress launched the browser process to run the tests and it sat there, blank.

Having seen this movie before, I took a guess and opened Activity Monitor.\nAs expected, the Cypress process was using 100% of the CPU.\nEventually, CPU usage dropped to zero, but Cypress still showed nothing.\nI knew that errors in the plugin loading stage sometimes (maybe always?) pass silently, and I knew that we only had one plugin, and I knew that it was serializing a lot of data.

A quick check and... yes, every Example in the system contains a circular reference to its Lesson.\nBut then why did this cause 100% CPU usage for several seconds?\nShouldn't Node have errored immediately after seeing the circular reference?\nWhat probably happened is: the curriculum object was so large that it took multiple seconds to get to the point where it could even check for cycles!\n(As I write this, our course list contains 4,024 code examples, plus all of the accompanying text, all stored in fine-grained data structures with a lot of metadata.)

The fix

The bug is weird, but the fix is very simple.\nHere's the buggy code that creates a circular reference:

const example: Example = {\n  kind: "example",\n  id,\n  code,\n  expectedResult,\n  isInteractive,\n  lesson,\n}\n

We need to avoid putting the entire lesson object inside the example.\nInstead, we pull out only the property we care about, the ID:

const example: Example = {\n  kind: "example",\n  id,\n  code,\n  expectedResult,\n  isInteractive,\n  lesson: {\n    id: lesson.id\n  },\n}\n

Now there's no cycle and everything works again.\nA trivial solution to a complex problem implicating several subtle topics.\nTo summarize:

TypeScript allows excess properties on objects, which will exist at runtime even though they're not shown in the static types.
TypeScript's generated JavaScript code doesn't strip those excess properties, so they'll still exist at runtime.
This allows us to write code where the types have no cycles, but the objects at runtime do have cycles.
Which will cause an exception when trying to serialize a circular object into JSON.
Which can cause silent failures if your third party tools, like test runners, aren't careful about handling errors.

Where do we place blame here?\nTypeScript's design decisions, including this one, allow it to be flexible enough to interact with existing code on the web.\nIt's hard to argue against that; it's a major reason that we use TypeScript in the first place.\nAnd JSON.stringify is correct to error on cyclic data because that data is unrepresentable in JSON.

Cypress' silent error handling, on the other hand, was a genuine bug.\nBut suppose that it had surfaced the error, "TypeError: Converting circular structure to JSON".\nWe'd still have to understand circular data structures, and we'd still have to understand the TypeScript design quirk that lets them show up even when the types don't allow them.

That leaves us with this plan for debugging problems like this:

Know the subtleties of your tools, especially your programming language, so you can make reasoned guesses when you see surprising behavior.
Optional: only use perfect tools that will never have bugs, ever.\nThis will let you skip some steps, like using Activity Monitor to guess at what Cypress is doing, but you still have to do most of (1).

\n html 1587415508 Restoring React Reducer State Across Browser Sessions https://www.executeprogram.com/blog/restoring-react-reducer-state-across-browser-sessions

This is a story about how we used io-ts to save React useReducer state for later.\nSaved state may be restored months later, in a different browser, while both the React code and the data used by the reducer have continued to change over time.\nThe solution described here is in production on the site you're looking at.\nFirst, a short description of the application itself:

Execute Program is an interactive learning platform made up of courses (like TypeScript or Regular Expressions).\nEach course is made up of many lessons (like Generic function types or Character classes).\nAnd each lesson is made up of interactive code examples mixed with paragraphs of text.

As a user goes through a lesson, they reveal one example or paragraph after another.\nWe call those "steps": each code example is a step and each paragraph is a step.\nWhen they reach the last step, the client code tells the server that the user has finished the lesson.

Here's the problem:

A user starts a long lesson that might take 20 minutes to finish.
They close the browser tab for some reason.
The next day, they go back to the lesson.
Now they have to start from the beginning of the lesson.

Here's a video of our solution in action.\nWe advance through a paragraph and an initial, non-interactive code example.\nThen we reload the page and choose to resume the lesson.\nThen we reload the page again and choose to restart the lesson.

\n \n

Let's examine a series of possible solutions, beginning with the simplest.

Solution 1: Store the current step index.

React's useReducer manages large state objects with complex transitions.\nThe developer defines how the object will change in response to user actions.\nFor example, when we clicked the "Continue" button in the example above, our reducer incremented the step index by 1.

In our lessons, the reducer state contains the full step objects as retrieved from the API, including the actual code and paragraph text.\nIt also contains various metadata, including an index variable: a number representing the current step's index in the step array.\nWhen the user finishes a step – for example, by completing a code example – we increment the index and the UI shows the next step.

The first possible solution is to save the step index somewhere: maybe on the server, or maybe in the browser's localStorage.\nThis works... but only if our lessons never change.

Our lessons changes almost every day as we tweak them in response to user feedback.\nIf we insert, delete, or reorder steps, then the stored index will be wrong.\nThe user will get to a certain point in a lesson, then leave, then come back, and the lesson will resume at the wrong step.\nWe need a solution that always resumes to the correct point in the lesson.

Solution 2: Invalidate the step index when the lesson changes.

We can store the step index on the server or in the browser's local storage, but also have a way to invalidate it when the lesson content changes.\nThe easiest solution here is a hash over the entire lesson's content: all of the code examples and all of the explanatory text.\nWe store the hash alongside the step index.

Later, when a user returns to the lesson, we compare the stored hash against the current lesson hash.\nIf they match, we offer to resume the lesson.\nAny content change will result in a different hash, so we'll ignore the stored step index whenever the lesson changes.\nThis works... but it only restores the step index.\nWhat about the other metadata that the client gathers during the lesson?

We care about that metadata!\nFor example, we want to know how many attempts the user made at each code example.\nAggregating those numbers can show us when certain examples are too difficult.

We need a solution that preserves that metadata in addition to the step index.\nOne idea is to pull out the pieces of the reducer state that matter and store only them.\nThat would include the exact content of the lesson (or a hash of it), the number of attempts the user made at each code example, etc.

Solution 3: Store the entire reducer state.

As it turns out, we care about most of the data in the reducer state; otherwise we wouldn't be tracking it!\nThe easiest solution is to store the entire reducer state, then restore it later when the user resumes the lesson.

Storing the reducer state brings up another data mismatch problem.\nLast time, the mismatch was in the lesson content.\nNow, the mismatch is in the schema or structure of the reducer's state object: the actual keys in the state, and the particular types of data in those keys.

Here's the problem that will occur:

The user does part of a lesson.
We store the reducer state.
A month passes.\nWe make code changes that change the structure of the reducer state: adding and removing keys, changing data types, etc.
The user comes back and resumes the lesson.
We restore the reducer state.
Now the user is running a new version of our app, which expects the new reducer state data, but we just restored the old reducer state data.
The app crashes (or worse).

We need a solution that can ignore any saved reducer state that was created by an older, incompatible version of the client-side code.

Solution 4: Version the reducer state.

The obvious solution is to add a version number to the saved state data.\nWhen we make an incompatible change to the reducer state's structure, we change the version number.\nWhen restoring an old reducer state, we compare its version number against the current code's version number.\nIf they don't match, we ignore the saved state and start from the beginning of the lesson.

Unfortunately, humans are bad at remembering to do things.\nIt's likely that we'd change the reducer state's structure at some point, but forget to change the version number.\nThis is especially likely because the state contains the lesson content itself, which comes from our API.\nWe'd have to remember to increment the version number when changing the state, but also whenever we change any of the related API endpoints.\nThat's too error-prone.

We could automate the version number.\nFor example, we could hash the source files that define the relevant parts of the API, as well as the file that defines the reducer state, and use that hash as the "version number".\nBut that has its own problems.

First, even trivial changes like adding or removing whitespace would invalidate all lesson states for all users.\nSecond, what happens if we break one of those API files into two different files?\nIf we forget to add those two new files to our hashing scheme, then the system won't know to invalidate saved states when they change.\nHashing the source files would only provide the illusion of safety; in reality, it's probably more error-prone than a manual version number.

We need a solution that can detect changes to the structure of the state data only when that structure genuinely changes, and regardless of why it changed.

Interlude: An introduction to io-ts.

Execute Program is written in TypeScript.\nIt already uses io-ts to do runtime validation of API data.\nFor example, when an API request comes in, we use io-ts to check that all of the keys are present; that keys that should be numbers are numbers; that arrays are arrays; etc.

Our reducer problem is similar to the API problem.\nIn an API, we have network data coming in from another computer.\nIt could be coming from a malicious attacker, or it could be coming from a future version of ourselves who made a mistake and sent the wrong data to our own API.\nWe want to detect those problems and throw an error.

In our reducer state problem, we also have data coming in from "another" computer.\nIn this case, that computer is the user's own computer from some point in the past: the state was saved a month ago and it's being resumed now.\nLike with the API, we need to check the structure of the data: make sure that all expected keys are present; that keys that should be numbers are numbers; etc.\nWe can use io-ts to solve this, just as we do with our API.

Here's a quick primer on how io-ts works.\nSuppose that our API has a register endpoint for registering new accounts.\nIt accepts an email address.\nWe can define an io-ts codec that validates the incoming data:

import * as t from 'io-ts'\nimport { isRight } from 'fp-ts/lib/Either'\n\nconst register = t.strict({\n  email: t.string,\n})\n

Then, we can use it to check for whether some data matches our codec:

// This returns true.\nisRight(register.decode(\n  {email: 'amir@example.com'}\n))\n\n// Each of these returns false.\nisRight(register.decode(\n  {email: 1234}\n))\nisRight(register.decode(\n  {email: undefined}\n))\nisRight(register.decode(\n  {userEmail: 'amir@example.com'}\n))\nisRight(register.decode(\n  {}\n))\nisRight(register.decode(\n  null\n))\n

This is how Execute Program checks API data.\nWe define io-ts codecs for every API endpoint.\nThen the server uses decode and isRight to validate all incoming and outgoing data.

Some of our API data is complex: objects nested in arrays nested in other objects nested in yet other objects, etc.\nWe break those complex io-ts codecs into smaller, more manageable pieces, which are sometimes shared between multiple endpoints.

There's one more important fact about io-ts.\nWe're in TypeScript, so we want static types for our data.\nFor our register codec above, that might look like:

type Register = {\n  email: string\n}\n

But that duplicates our io-ts codec.\nInstead of writing the type manually, we can extract the exact type above from the io-ts validator:

type Register = t.TypeOf<typeof register>\n

That will look quite arcane if you're new to TypeScript.\nBut this pattern always looks the same when extracting a static type from an io-ts codec, so it becomes routine with with some TypeScript and io-ts experience.\nOne short line of code using t.TypeOf can replace hundreds of lines of duplicated type definitions.

(Full disclosure: doing this sometimes makes type error messages more confusing, but the details of that are out of scope here.\nIn some situations, we do define the types explicitly, effectively duplicating the io-ts codec definitions.\nIt's annoying, but the type system will keep us honest: it will error if our types don't match the codec's structure.\nThese definitions are used all over the system but change rarely, so a bit of extra code in a complex codec is often a good trade-off to get better error messages.)

To summarize:

io-ts lets us validate data at runtime, ensuring that it has the structure that we expect.
We can extract static types from our io-ts codecs, so we don't have to define the types twice unless we choose to.

Solution 5: Validate the reducer state with io-ts.

Now we can talk about our chosen solution for the lesson resuming problem.

We define an io-ts codec that describes the structure of the reducer state.\nRemember that the reducer state contains the steps (the code examples and paragraphs), which come from the API.\nOur entire API is defined using io-ts, so we can reuse all of those codecs.\nExcluding imports and whitespace, the io-ts codecs for our reducer state are 66 lines long.

If we ever change the reducer state's structure, we'll be forced to update the io-ts codec.\nThat's because the TypeScript type system checks all of this: the io-ts codec must match the reducer state types, and the reducer code in our React components must also match those types.

Now we can formulate a solution to the problem of deciding whether a lesson state came from an old, incompatible version of the application:

Deserialize the JSON-encoded state.\nIf it's not legal JSON, abort.
Check the JSON against our reducer state io-ts codec.\nIf it doesn't conform to the io-ts codec, then it's from an older, incompatible version of the code.\nAbort.
If we get here then the state is compatible!

Now we have to solve the second problem: what if the lesson content has changed?\nFortunately, the reducer state contains all of the steps (the code examples and paragraphs) as they existed when the state was saved.\nThe client also has the current examples and paragraphs: it's showing a lesson, so it just got all of the lesson content from the API.\nWe can compare the two to decide whether the content has changed since this state was saved:

Get the steps array from the reducer state.
Use the API's io-ts codecs to "narrow" those steps to only the keys that exist in the API.\nThis removes the metadata fields added by our reducer.\n(In addition to validating data, io-ts codecs can narrow data down, which removes any object keys that are in the data but aren't part of the codec.)
Use lodash's isEqual to do a deep comparison of the saved state's steps and the current API steps.
If they're not equal, then the content has changed since this state was saved, so abort.

This is a full solution to the problem!\nThe type system ensures that we never create invalid reducer states in our normal React code.\nThe io-ts codecs ensure that we never restore an old state with the wrong structure.\nAnd checking the saved steps against the current steps ensures that we don't confuse the user or the app by restoring to the wrong point.

Here's the function that loads the state.\nIt takes a string that (hopefully) contains a JSON-serialized state.

export function loadSerializedState(\n  apiSteps: readonly ApiStep[],\n  serializedModelState: string\n): State | undefined {\n  return validateSteps(\n    apiSteps,\n    validateStateType(\n      deserializeJson(serializedModelState)\n    )\n  )\n}\n

The three functions called here are quite short.\nExcluding imports and comments, all four functions sum to 38 lines of code.

Each of the three functions called here will return undefined if something doesn't match, and each of them passes undefineds along.\nThat allows us to write the top-level loadSerializedState function in this simple way.

If loadSerializedState returns an object, then we know it's a valid reducer state that corresponds to the current lesson content and the current client-side code.\nWe feed it into useReducer, and React uses it without knowing or caring that it was created a month ago!

It's important to note that this is a conservative solution.\nSuppose that we make a typo in the lesson content: we write "proimse" instead of "promise".\nWhen we fix that typo, the current lesson content will no longer match any of the saved states for that lesson.\nEvery saved state for that lesson is immediately invalidated.

In the future, we may try to detect trivial content changes and allow those saved states to be restored.\nBut the perfect is the enemy of the good, and we get most of the benefit without doing any deep content comparison.

Where do we store the state?

Now that we can save and restore our reducer state, there's one more question: where do we store it?\nThe initial plan was to store it in the browser's local storage.\nBut that has a big problem.\nIf a user begins a lesson on their phone, then later tries to resume it on their laptop, the state won't be there.\nThe laptop's browser can't see the saved state on the phone.

Our actual solution is to store the state in our server-side database.\nThis feels wrong at first glance: we're storing a client-side React component's state as an opaque JSON blob in our server-side database, and we're doing that knowing that some of those JSON blobs will go out of date and be unusable in the future.\nBut io-ts and the type system keep us honest here!\nAnd this is a convenience feature, so we can always change it later, even if that means throwing away all of the saved states.\n(When a user finishes a lesson, that record is stored in a separate part of the database.)

A simple solution

It took a lot of words to describe this solution, but that's because there was a lot of background to explain.\nIn reality, I went almost directly to the io-ts solution because I could see that this problem was analogous to the API validation problem.

This code change was quite simple: it added 470 lines of code containing 1,102 words.\n(This article contains almost three times as many words!)\nThose 470 lines include the new modal UI for resuming, the new API endpoint, the API server handler, the database migration, and the database model class, all of which would exist with or without this solution based on io-ts.

Our tools are a good match for our problems, which allows us to solve problems like this smoothly:

React's useReducer represents states as simple JavaScript objects, so we can serialize them to JSON and bring them back later.
Defining our API's structure with io-ts makes it easy to validate objects that contain API data.
TypeScript statically guarantees that our system's data all has the correct shape.\nIf io-ts says that the restored state object is correct, we can trust TypeScript to keep other parts of the system honest.

If you're interested in trying Execute Program to see this in action, we recommend the TypeScript courses or the Regular Expressions course.\nPlease use the feedback item in the menu to tell us what you think!

\n html 1586983508 Problems With TypeScript in 2020 https://www.executeprogram.com/blog/problems-with-typescript-in-2020

TypeScript is very good.\nWe ported our frontend and backend to it with no regrets.\nWe have a much smaller test suite than we ever could've had in a dynamic language.\nAll of this is great.\nStill, there are downsides to TypeScript, and we should be honest about them too.

JavaScript

TypeScript is a superset of JavaScript, and JavaScript is a mess.\nIt was designed in 10 days, then it evolved haphazardly for almost two decades, then it was finally cleaned up in the last few years.\nA lot of the warts are still there and will always be there. For example:

1 * '1' returns 1 instead of an error.
({name: 'Amir'}).age returns undefined rather than erroring.
x && x + 1 looks like it means "add 1 to x if it's not null or undefined"; but when x is 0 it returns 0, not 1.

TypeScript can't fix all of JavaScript's warts.\nBut having used JavaScript since the 90s, I'm amazed at how well TypeScript did its dirty, improbable job.\nThe first two code examples in the previous list are illegal in TypeScript.\nBy adding a linter, the third one becomes illegal too.\nThose examples are simple, but TypeScript has saved us from many complex, subtle bugs that are too complex to show here.

The TypeScript team has been working to standardize new JavaScript features that let them increase safety further.\nFor example, the new nullish coalescing operator mitigates JavaScript's design defects around the falsiness of 0 and the empty string, "".

Overall, the JavaScript underneath TypeScript is still frustrating in many situations, but TypeScript mitigates many of the worst frustrations.\n(A past version of myself would be very surprised to see this future!)

Compiler bugs

For decades, programmers joked that "It's never a compiler bug."\nCompilers weren't perfect, but "compiler bugs" almost always turned out to be the programmer misunderstanding the language.

When used as a long-lived watcher process, TypeScript has broken this trend: it's buggy.\nThe compiler tends to hit bugs when files are deleted or renamed.\nWe have a special script that notices those events and restarts the compiler.\nBut our script is imperfect, so we still end up restarting the compiler manually.\nLast Saturday, I manually restarted the TypeScript compiler dozens of times as it encountered multiple bugs over and over again.

Sometimes the compiler doesn't realize that a file was deleted, so it produces incorrect errors.\nFor example, if file A imports file B, and we delete both, it will sometimes complain that file A is trying to import B, which doesn't exist.\nBut A doesn't exist either, so it shouldn't be causing an error!\nOther times, the compiler "successfully" compiles when there are actually type errors.\nIn all of those cases, killing and restarting the compiler makes the errors go away, which is an unambiguous sign that it's a compiler bug in the watcher's state management.

(Webpack's watcher also has similar bugs around file deletion and renames.\nThe easiest workaround for both TypeScript and Webpack is to restart all of the tools when a file is renamed or deleted, then wait as they all start back up.\nI feel embarrassed while waiting for those restarts.)

In the compiler's defense, this all seems to be limited to the watcher.\nI've never seen a bug in the compiler when it's cold booted, as it is during a production deploy or a CI run.\nAnd TypeScript doesn't seem worse than most other long-running development tools.\nAs any long-time IDE user knows, any development tool that runs for a long time will break.

Unfortunately, booting the TypeScript compiler is very expensive, so we have no choice but to keep the compiler running between changes.\nThis is in contrast to some other static languages like Go and Reason, which can compile systems the size of Execute Program in the neighborhood of 100 times faster than TypeScript.\nThose compilers are fast enough to be booted as-needed and then terminated.\nNo need for a long-lived watcher.

The Node/NPM ecosystem

TypeScript is a superset of JavaScript, so it's closely connected to the Node and NPM ecosystems.\nLike any large ecosystem, the Node and NPM ecosystem is a mixed bag.\nThere are so many packages; it's great!\nBut some of them are very good and others are less good.

TypeScript allows us to write types for existing JavaScript libraries.\nPeople (often not a package's original authors) publish these type definitions in a central repository.\nOn GitHub, that repository says "The repository for high-quality TypeScript type definitions."\nThere's a bit of aspiration in that description.\nSometimes it's true and the type definitions are high-quality!\nSometimes they're not.

As one example, we recently did a major version upgrade of the database library that underpins our own database library.\nThat made some of our valid code fail to type check.\nWorse, it allowed other code to type check even though it had major type errors.\nFrom our perspective, the "upgrade" was a step backward.\nBut we don't have a choice if we want to continue to get security updates, and continue to upgrade other packages that depend on the database library.

None of that happened because TypeScript is bad; it happened because the type definitions for that library are wrong.\nFortunately, our recent experience with that database library is an outlier.\nMost of the time, everything goes fine.\nStill, this is an annoyance that's relatively unique to TypeScript because it's a type system layered over an existing dynamic language.

It's hard to say whether the Node+NPM+TypeScript ecosystem is "better" or "worse" than, say, Ruby's or Python's.\nHowever, it's definitely far more complex, and it requires a more careful approach.\nSometimes, there will be 50 packages that claim to do the thing that we need.\nSometimes, one of them has more downloads than the other 49 combined.\nSometimes, the best solution is the ninth-most-popular.

Overall, less-experienced programmers face an uphill battle in deciding which packages are worth using.\nAdding TypeScript increases that burden: now we have to analyze all of the library options, and we also have to analyze the presence and quality of third-party type definitions.\nArguments for or against packages may be written by JavaScript users, so they won't be considering the quality of the library's TypeScript type definitions.

These problems don't outweigh the benefits.\nCompiler bugs are annoyances, but they've never affected our CI or production environments.\nEcosystem complexity is a danger that can be navigated with forethought and by learning from past mistakes.\nI mention these problems to balance out the praise in earlier articles about TypeScript, but for us the trade-off is unambiguously in TypeScript's favor!

Gary Bernhardt

\n html 1586810708 Are Tests Necessary in TypeScript? https://www.executeprogram.com/blog/are-tests-necessary-in-typescript

We ported our React frontend from JavaScript to TypeScript, then did the same for our Ruby backend.\nIt was a success; large classes of mistakes became impossible.\nYou might wonder whether we could have achieved these results with full test coverage instead.

Type systems aren't good at everything.\nOne common example is: type systems usually can't stop us from accidentally inverting a conditional.\nIn most cases, switching the "if" and "else" won't cause a type error.\nFor that, we have tests.

In Execute Program, we always write tests around subsystems that either (1) are critical to the product or (2) contain a lot of details that can't be statically checked.\nBilling is covered by tests because we don't want to charge someone incorrectly.\nOur progression course model is covered by tests because it contains a lot of conditional logic that the type system can't check.

We don't want tests covering most of our React components, though.\nThat wouldn't help with the main difficulties in writing components:

We might pass props around incorrectly.\nTypeScript already solves this problem for us almost completely.
The components might use the API incorrectly.\nAgain, we've solved this problem with TypeScript.
The components might look wrong when rendered.\nTests are very bad at this.\nWe don't want to go down the esoteric and labor-intensive path of automated image capture and image diffing.

It may still be a good idea to cover all of our components, even if our main component problems aren't solved by normal tests.\nWe should do a cost/benefit analysis to find out what we'd be "paying" to get those tests.

First, what's our starting state?\nOur entire system is 24,065 lines including all current tests, with none of those tests covering the components directly.\n(Some of the components are partially exercised through browser-driving tests.)

Assuming a 2:1 test:code ratio for full coverage, covering our 9,000 lines of component code with unit tests would require 18,000 lines of test code.\nAdding 18,000 lines of client tests would almost double the total amount of code that we maintain.\nIn practice, we already have a low defect rate in our TypeScript frontend, so there's no reason to pay that maintenance cost.

Although we don't test components directly, there are some other parts of the frontend code that are tested directly.\nFor example, we have some high-value tests around some React reducers because they're tricky and full of conditionals.\nThose tests add up to around 300 lines, giving us a test:code ratio of about 1:30 for client code.

We also don't have tests covering our backend API handlers.\nOnly a few of those handlers have any conditionals at all; they mostly translate API requests into calls to other subsystems.\nWe push conditional logic down into those subsystems and test it there.\nAs usual, TypeScript ensures that the handlers are properly wired up to the lower subsystems.

The subsystems below the API handlers do all kinds of different things.\nWe have subsystems for tracking which lessons a user has unlocked; for computing statistics about their finished lessons; for managing subscriptions and billing; for deciding what content users have access to; etc.\nAll of those subsystems are fully tested.\nThat gives the server a roughly 1:2 test:code ratio: two lines of production code for every line of test.

All of this is to say: there are places where tests are necessary to achieve confidence in the system.\nBut any production web application will have large regions where we can gain confidence via the type system and avoid writing almost all tests.

We have 19,498 lines of production code covered by 4,567 lines of tests.\nThat gives an actual test:code ratio of about 1:5 overall.\nIt feels about right.

Let's imagine an alternate version of Execute Program with a 2:1 test:code ratio.\nNaively covering all 19,498 lines of our production code with 2:1 tests would require 38,996 lines of test code, increasing our total line count from 24,065 lines to 58,494 lines.\nThat's 243% as much code for very little additional benefit.

To answer the question that we started with: could we have achieved these results by forgetting TypeScript and fully covering the app with tests?\nThe most literal answer is: yes, we might be able to get to the same low defect rate, but that ignores all other trade-offs in play here.\nWe'd be maintaining an extra 150% or so more code, depending on our final test:code ratio.\nWe'd also spend a lot more time writing and maintaining test code, rather than building features.

Types and tests are not equivalent at all; they give very different kinds of confidence.\nTests can never provide an unbroken chain of checks, from the database all the way to the frontend React props, guaranteeing that all of the data has the right shape.\nTypes can't (usually) tell us when we accidentally put a "!" in front of a conditional.\nFocusing on one or the other means sacrificing quality, work efficiency, or both.

Gary Bernhardt

\n html 1586378708 Porting to TypeScript Solved Our API Woes https://www.executeprogram.com/blog/porting-to-typescript-solved-our-api-woes

We ported our React frontend from JavaScript to TypeScript, but left the backend in Ruby.\nEventually, we ported the backend to TypeScript too.

With the Ruby backend, we sometimes forgot that a particular API property held an array of strings, not a single string.\nSometimes we changed a piece of the API that was referenced in multiple places but forgot to update one of those places.\nThese are normal dynamic language problems in any system whose tests don't have 100% test coverage.\n(And it will still happen even with 100% coverage; it's just less likely.)

At the same time, these kinds of problems were gone from the frontend since porting it to TypeScript.\nI had more experience with backend development, but I was making more simple errors in the backend than the frontend.\nThat was a sign that porting the backend was a good idea.

I ported the backend from Ruby to TypeScript in about two weeks in March of 2019.\nIt went well!\nWe pushed it to production, which was then in a closed beta, on April 14, 2019.\nNothing blew up; users didn't notice.\nHere's a timeline of the run-up to the backend port, plus the time immediately after:

I wrote an unusual amount of custom infrastructure during that port.\nWe have a custom 200-line test runner; a custom 120-line database model library; and a larger API router library that spans the frontend and backend code.

Of our custom infrastructure, the router is the most interesting piece to discuss.\nIt wraps Express, enforcing the API types that are shared between the client and server code.\nThat means that when one side of the API changes, the other side won't even compile until it's updated to match.

Here's the backend handler for the blog post list, one of the simplest in the system:

router.handleGet(api.blog, async () => {\n  return {\n    posts: blog.posts,\n  }\n})\n

If we rename the posts key to blogPosts, we get a compile error ending with the line below.\n(The actual object types are removed from the error message to keep it short here.)

Property 'posts' is missing in type '...' but required in type '...'.\n

Each endpoint is defined by an api.someNameHere object, which is shared between the client and server.\nNotice that the handler definition doesn't name any types directly; they're all inferred from the api.blog argument.

This works for trivial endpoints like blog above, but it also works well for complex endpoints.\nFor example, our lesson API endpoint has a deep key .lesson.steps[index].isInteractive, which is a boolean.\nAll of these mistakes are now impossible:

If we try to access isinteractive on the client or to return it from the server, that won't compile; it has to be isInteractive, with a capital "I".
If the server returns a number for isInteractive, it won't compile.
If the client stores isInteractive in a variable of type number, it won't compile.
If we change the API definition itself to say that isInteractive is a number rather than a boolean, then neither the client nor server will compile until they're fixed.

None of that involves code generation; it's done using io-ts and a couple hundred lines of custom router code.

There's overhead in defining these API types, but it's not difficult.\nWhen changing the API's structure, we have to know how the structure is changing.\nWe write our understanding down in the API type definitions, then the compiler shows us all of the places that have to be fixed.

It's difficult to appreciate how valuable this is until you've used it for a while.\nWe can move large sub-objects in the API from one place to another, rename their keys, split one object into two separate objects, merge multiple objects into one new one, or split and merge entire endpoints, all without worrying about whether we missed a corresponding change in the client or server.

Here's a real example.\nI recently spent around 20 hours redesigning Execute Program's API over four weekends.\nThe entire API structure changed, totaling tens of thousands of lines of diff across the API, server, and client.\nI redesigned the server-side route definition code (like handleGet shown above); rewrote all of the type definitions for the API, making huge structural changes to many of them; and rewrote every part of the client that called the API.\n246 of our 292 source files were modified in this change.

Throughout most of the redesign, I relied only on the type system.\nIn the final hour of the 20-hour process, I started running the tests, which mostly passed.\nAt the very end, we did a full manual testing pass and found three small bugs.

All three bugs were logic errors: conditionals that accidentally went the wrong way, which type systems usually can't detect.\nThe bugs were fixed in a few minutes.\nThat redesigned API was deployed a few months ago, so it served you this blog post (and everything else on Execute Program).

(This doesn't mean that a static type system will guarantee that our code is always correct, or guarantee that we don't need tests.\nBut refactoring becomes much easier.\nWe'll talk about the larger question of testing in the next post.)

There is one place where we do code generation: we use schemats to generate type definitions from our database structure.\nIt connects to the Postgres database, looks at the columns' types, and dumps corresponding TypeScript type definitions to a normal ".d.ts" file used by the rest of the application.

The database schema type file is regenerated by our migration script on every migration run, so we don't do any manual maintenance on those types.\nThe database models use the database type definitions to ensure that application code accesses every part of the database correctly.\nNo missing tables; no missing columns; no putting a null in a non-nullable column; no forgetting to handle null in columns that are nullable; all statically verified at compile time.

All of that together creates an unbroken statically typed chain from the database all the way to the frontend React props:

If a database column's type changes, other server-side code (like the API handlers) won't compile until everything is updated to match.
If the server-side API handlers don't match the client-side API consumers, one or both won't compile.
If the client-side React components don't match the data coming out of the API, they won't compile.

Since finishing this port, I don't remember any API mismatch making it past the compiler.\nWe've had no production failures due to mistakes where the two sides of the API disagree about the shape of the data.\nThis isn't due to automated testing; we don't write any tests for the API itself.

These guarantees are wonderful: we can focus on the parts of the app that matter!\nI spend very little time wrangling types – far less than I spent chasing down confusing errors that propagated through layers of Ruby or JavaScript code before causing a confusing exception somewhere far away from the original source of the bug.

Here's a timeline of our development since the backend port.\nWe've had a lot of time and new code to evaluate the results:

We haven't discussed a common objection to any post like this: couldn't you get the same result by writing tests?\nAbsolutely not, which we'll cover in the next post!

Gary Bernhardt

\n html 1586205908 Porting a React Frontend to TypeScript https://www.executeprogram.com/blog/porting-a-react-frontend-to-typescript

The beta version of Execute Program was written in Ruby and JavaScript.\nThen we ported all of it to TypeScript in multiple steps.\nThis is the story of the frontend port, which was the first step.

In Execute Program's original JavaScript frontend, I often made small mistakes.\nFor example, I'd pass the wrong prop names to a React component, or leave a prop out, or pass the wrong kind of data.\n(Props are pieces of data sent as arguments to a React component.\nIt's common for a component to pass some props to one of its sub-components, which pass them on again, etc.)

This is a big problem with dynamic languages like JavaScript and Ruby.\nI've been learning to mitigate it for 15 years.\nHere I am talking about it way back in 2011.\nMitigations like the ones discussed there do help, but they don't scale well as a system grows, and there's no safety net for when we forget them.

I thought that 15 years was enough; I wanted to return to static type systems, where these mistakes are impossible.\nThere were a few options: Elm, Reason, Flow, TypeScript, PureScript.\n(That's not an exhaustive list.)\nI decided on TypeScript because:

TypeScript is a superset of JavaScript, so porting to it is easy.\nReversing a port is even easier: delete the type definitions; now we have JavaScript again.
The TypeScript compiler is written in TypeScript and distributed as compiled JavaScript, so we can run it in our web app.\nOur TypeScript course does exactly that: we evaluate users' TypeScript code in the browser to avoid network latency.
This one is particular to our business, but: TypeScript is more popular than the other options.\nThat means more people want to learn TypeScript from courses like ours.\nWriting Execute Program itself in TypeScript was a good way to set us up for writing a TypeScript course.

Porting the frontend JavaScript code to TypeScript took about 2 days in October of 2018.\nHere's a plot showing how much code we had in each language leading up to, and immediately after, that port.

That was pre-beta, so the system was still small: about 6,000 lines.\nThe period after the port is missing here; we'll expand on that in future posts.

After that port, React prop problems disappeared.\nWe'll look at a couple examples, starting with an easy one.\nHere's the code that renders the "Continue" button that appears after every paragraph of text in our courses:

<Button\n  autofocus={true}\n  icon="arrowRight"\n  onClick={continue}\n  primary\n>\n  Continue\n</Button>\n

The type of that Button component's props is shown below.\nWhen reading a property type like autofocus?: boolean, "autofocus" is the name of the property; "?" means that it's optional; ":" separates the property name from its type; and "boolean" is the type.\nThe last property type, onClick, means "a function that takes no arguments and returns nothing".\n(If TypeScript's function type syntax is unfamiliar, you can get a comprehensive overview in our lesson on TypeScript's function types.)

type ButtonProps = {\n  autofocus?: boolean\n  icon?: IconName\n  primary?: boolean\n  onClick: () => void\n}\n

What happens if we change the "autofocus" prop from true to 1?\nWe're now passing a number value where the type system expects a boolean.\nLess than a second later, the compiler prints the error below.\n(Some irrelevant details have been removed here; we'll do that for all of the errors in this series of articles.)

src/client/components/explanation.tsx(13,27):\n  error: Type 'number' is not assignable to type 'boolean | undefined'.\n

The offending code also turns red in vim.\nI fix it and the red goes away.\nFixing the mistake takes seconds.\nIn Ruby or JavaScript, I might spend minutes manually testing the app and rummaging around in its state to find out what happened.\n(I could also rely on automated tests, but we cover the issue of tests vs. types in another post.)

That integer-to-boolean change was a simple and low-stakes use of the type system.\nButton's icon property shows more advanced use.\nHere's the Button invocation again:

<Button\n  autofocus={true}\n  icon="arrowRight"\n  onClick={continue}\n  primary\n>\n  Continue\n</Button>\n

It looks like the icon prop is just a string: "arrowRight".\nAt runtime, in the compiled JavaScript code, it will be a string.\nBut in the ButtonProps type shown above, we defined it as an IconName, which is defined elsewhere.

Let's see what the type does before we look at its definition.\nSuppose that we change the "icon" prop to "banana".\nWe don't actually have an icon named "banana".

<Button\n  autofocus={true}\n  icon="banana"\n  onClick={continue}\n  primary\n>\n  Continue\n</Button>\n

Less than a second later, the TypeScript compiler rejects that change:

src/client/components/explanation.tsx(13,44):\n  error: Type '"banana"' is not assignable to type\n    '"menu" | "arrowDown" | "arrowLeft" | ... 21 more ... | undefined'.\n

The compiler is saying that "icon" can't be any arbitrary string; it has to be one of the 24 strings that we've defined as icon names.\nThe compiler will reject any change that leaves us referencing a non-existent icon; it's not a valid program and can't even begin executing.

There are multiple ways to implement the IconName type.\nOne is to write a type that explicitly lists all of the possible icon names.\nThen we'll have to keep the icon names in sync with their image files on disk.\nThat type might look like:

type IconName =\n  "menu" |\n  "arrowDown" |\n  "arrowLeft" |\n  "arrowRight" |\n  ...\n

In English: "a value of type IconName is statically guaranteed to be one of the strings specified here, but not any other string."\n(This type is a combination of two topics covered by Execute Program lessons: literal types and type unions.)

Our IconName isn't defined as a simple union of literal string types.\nKeeping a list of icon names in sync with a list of files is boring work that we can make the computer do!\nInstead, our icon.tsx file looks like this:

export const icons = {\n  arrowDown: {\n    label: "Down Arrow",\n    data() {\n      return <path ... />\n    }\n  },\n\n  arrowLeft: {\n    label: "Left Arrow",\n    data() {\n      return <path ... />\n    }\n  },\n\n  ...\n}\n

The actual SVG <path /> tags are right inside the source code, in an object keyed by the icon's name.\n(It's also possible to do this without inlining the SVG into a source file.\nFor example, we could use some Webpack tricks to keep the images in their own files, but still have a guarantee that every icon in the list also exists on disk.\nSo far, this simpler solution has worked well for us.)

By defining the icons in this way, we can extract a union type of their names automatically with one line of code:

export type IconName = keyof typeof icons\n

(In English, you can think of that type as saying "whenever something has the type "IconName", it must be a string that matches one of the keys of the icons object.)

That's it; there's no other type-level work required.\nThe rest of the code is just a straightforward Icon React component that looks up an icon in a list and returns its SVG path.\nThere are no explicit TypeScript types in that function; it looks like pure JavaScript, though it's still type-checked.\nHere's a minimal version with all unrelated details stripped away:

export function Icon(props: {\n  name: IconName\n}) {\n  return <svg>\n    {icons[props.name].data()}\n  </svg>\n}\n

Now we can drop new icons into the "icons" list by putting the SVG tags in that source file.\nWhen we do that, the icon becomes available for use in the Button component, as well as any other part of the system that accepts an icon name.\nIf we delete an icon from the list, every part of the system that references it instantly fails to compile, ensuring that we have no stale icon references that can cause errors at runtime.

These examples are simple by static type standards, but I think that they illustrate how much low-hanging fruit there is in a web application.\nMost of an application's code doesn't involve advanced type system features; it's simple stuff like "make sure that we're passing the right props" and "make sure that our icons actually exist."

We do this kind of thing all over the system.\nSome more examples:

We have a Note component used throughout the system.\nIt has a tone prop to determine the style of the note: "info", "warning", "error", etc.\nIf we retire one of those tone options, we'll remove it from the union type, and all Note uses that referenced that tone will error until we update them.
Every URL that we link to is statically guaranteed to exist.\nWhen we rename or delete a URL, every component linking to it fails to compile until we update them to match.
When we link to those URLs, the type system ensures that we fill in any holes in the URL.\nFor example, the path "/courses/:courseId/lessons/:lessonId" has two holes, "courseId" and "lessonId".\nIf we try to link to that path but forget to supply a "courseId", then the code won't compile.
Every API request that we make on the client is statically guaranteed to match the payload structure of the corresponding server-side API endpoint.\nIf we rename a property in an endpoint, even deep inside a nested API object, then any code referencing that endpoint property will fail to compile until we update it to match.\n(We cover this in another post.)

Problems like these come up often in programming, especially in dynamic languages, but can be statically prevented without writing any automated tests and without doing any manual testing.\nSome of them take work; our API router verification was tricky to write.\nBut a lot of them are easy.\nThe one-line "IconName" type above really is the entire solution to the problem; it will work if you copy it into a TypeScript file.

Porting our frontend code to TypeScript was just the beginning.\nWe've since ported the backend from Ruby to TypeScript, then grown and maintained it for nine months after the port.

Gary Bernhardt

\n html 1572558308 Why You Should Learn SQL https://www.executeprogram.com/blog/why-you-should-learn-sql

The SQL language is old, strange, and important.\nWe'll take those in reverse order:

SQL is important.\nUnlike many other types of databases, SQL databases can give strong guarantees of data correctness.

If we have a regular, user-facing bug in our application, some users will see an error.\nWhen we fix the bug, the users stop seeing the error and everything is fine.

But if we have a bug that inserts incorrect data into the database, then fixing the bug doesn't fix the problem.\nEven after the fix, the database is full of incorrect data, so we have to do something to clean it up.\nHere's a concrete example:

We never expected orders to have a null phone_number, but a temporary application bug inserted some nulls there.\nNow our order history page is rendering those orders incorrectly.\nThe business reports used by the management team throw errors because they don't expect nulls, so the business's cashflow is now invisible.\nOur analytics continue to run, but they produce incorrect numbers because every order with a null phone_number is defaulting to country code 1 (the USA and Canada).

There's no way to retroactively get correct phone numbers for all of those orders!

We have two possible ways to mitigate this problem once it's happened.\nFirst, we can adapt the rest of the system to correctly handle the nulls.\nThat's a lot of work to handle a case that was never supposed to happen; we're only considering this because of a bug we already fixed.

The second option is to make up fake phone numbers for all of the affected orders.\nSaid another way: we can intentionally put incorrect data into the database, complicating attempts to contact those customers.

Neither of those options is desirable.\nHowever, there is a third option: tell the database that the phone_number is never allowed to be null.

In a SQL database like PostgreSQL, that non-nullability is guaranteed.\nIf we try to insert a null, the database rejects it.\nIf PostgreSQL allows us to insert a null in that column, it's a bug in PostgreSQL itself.\n(PostgreSQL is stable and well-tested; you should never expect to find a bug like this.)

OK: SQL is important because our data is important.\nIt's also very strange, but in ways that make it powerful.

Most popular languages are imperative: code starts at the top of the function and executes line by line until the bottom.\nThis is our default for a good reason: it's easy to think about how the code executes, at least within a single function.

SQL isn't like that at all.\nIn SQL, we describe the results that we want declaratively.\nFor example, in an imperative language we might say: "Loop over each cat. If a cat's age is over 5, add it to the oldCats array. Then return the oldCats array."\nIn SQL, we say something more akin to "Give me all of the cats whose age is over 5."

What happens when we execute these two versions of the code?\nThe imperative version says what the computer should do, step by step.\nIt's difficult for a compiler to optimize it because it's so rigid.

The SQL version doesn't say what order things should happen in; it just describes the result.\nThat lets the database optimize it aggressively.\nThe database will slice up the query into its smallest parts, reorder it as needed, throw some parts away if they're not needed, replace them with more efficient versions, etc.\nThis all happens without any input from us as the programmers, and it's guaranteed not to change the results.

In everyday situations, that optimization turns minutes-long queries into single-digit milliseconds.\nIn extreme cases, it turns years-long queries into seconds.

Compilers for imperative languages have optimizers too.\nBut SQL databases' optimizers are much more powerful because SQL lets us say what data we want, rather than how to get it.

OK: SQL is important and strange.\nIt's also old: it first appeared in 1974.\nThat definitely shows, so let's not kid ourselves.\nIn SQL, we say things like "ALTER TABLE users ALTER COLUMN email DROP NOT NULL".

In English, that means "allow users' email addresses to be null from now on".\nThe SHOUTING PARTS are SQL syntax, whereas "users" and "email" are parts of our data.\nThis shouting convention comes from THE OLD TIMES.

You can see that SQL is unusually wordy.\nJavaScript has 64 keywords: "if", "function", "for", etc.\nC has 34.\nSQLite has 140 keywords.\nBut SQLite is "lite", like its name says.\nPostgreSQL 11 has 760 keywords.\nThat's the exact number of words that you've read in this article so far, not including this sentence!

OK, so SQL is important, which makes it worth learning.\nBut it's also strange and old, which can make it offputting.

That's a great reason for us to teach Execute Program to teach you SQL.\nWe build complex knowledge models of tools like SQL, regular expressions, and TypeScript, which allows Execute Program to teach the topics in thousands of different orders depending on your strengths.\nUnlike most learning tools, you'll rarely go a full minute without completing an interactive code example.\nThose examples are then reviewed automatically, on exponentially increasing intervals, so you don't forget them.

You can try our first SQL lesson without creating an account.\nIt starts lightly, at the very beginning, and ramps up from there.\nOur favorite lesson comes late in the course, when you write a real SQL injection exploit to attack your own database.\nWe hope that you enjoy it!

Gary Bernhardt

\n html