Managing our focus during development can be a challenge.
Here's a common scenario: we're in the middle of working on user.ts when we find ourselves thinking: "I really need to go change this other thing in login.ts".
But it's rarely a good idea to interrupt ourselves during a complex change.
We have a lot of context in our heads that we don't want to lose.
There are two common ways to deal with this: we can write down a note that reminds us to do it later, or we can write a quick failing (or pending) test that forces us to do it later.
Both approaches work just fine. However, there's a third way that's superior to both in some situations: we can use the code itself as a to-do list. Specifically, we can use the linter to enforce our to-dos.
In Execute Program, we use ESLint with the "no-warning-comments" lint rule enabled:
module.exports = {
"rules": {
"no-warning-comments": ["error", {terms: ["xxx"], location: "anywhere"}],
...
},
...
}
This configuration makes any comment containing "XXX" an error. Lint errors cause our CI builds to fail, which means our XXX comments block deployment to production. As a result, we can now use XXX comments to mark critical problems that should never reach production.
(We could also use TODO comments instead of XXX by changing the ESLint configuration above. But our convention is that TODO comments are for longer-term deferred maintenance like "remove this old API endpoint" and XXX means "do not let this go to production". The particular string "XXX" isn't important here.)
This approach is often better than separate to-do lists and failing tests because:
In the rest of this post, we'll see how XXX comments have helped our development process. We'll review four real examples from the Execute Program codebase, ordered from simplest to most complex. These examples come from a feature branch that's diverged from our main branch by about 3,500 lines.
$ git diff main --stat | tail -1
40 files changed, 2448 insertions(+), 1155 deletions(-)
There are a lot of moving parts in this release, so it's crucial to keep track of all the changes we need to make as we develop!
1. Simplifying code
// XXX: Remove this module; it's trivial now.
We have a module that used to export several TypeScript types. On this branch, all but one of those types are now gone. We don't need this whole module to export one small type. The type can live in the subsystem's top-level module instead.
This change feels quick, so it's tempting to make it immediately. But it will be just as fast if we do it tomorrow. We don't have any special context in our heads that makes this change faster today.
On the other hand, we probably have a lot of context in our heads about the bigger change that we were making when we noticed this. Programming is highly dependent on short-term memory, and any distraction is a chance for us to forget details that may be important. Leaving an XXX comment only takes a few seconds, keeps our attention on the current task, and ensures that we'll remember to remove this module later.
2. Deferring a data format change
// XXX: This is inserting "Start Case" property names. They should be "camelCase".
While working on this branch, we needed to change a data format. Unlike the type definition change above, updating the rest of the code to match the new format will take a substantial amount of work. This change will also impact some other parts of the system, so we'd rather isolate it in a separate commit.
Like the module removal example, we don't want to distract ourselves. And the need for a separate commit makes deferring this change an even easier decision.
Sometimes, the fine-grained version control details also matter.
You might think: Why not make the change now, then tease the two changes apart after the fact with git add -p or an interactive rebase?
In this case, many of the affected lines are very close to lines that we've already changed.
Individual modifications from both of our changes would end up in the same diff hunks, which makes teasing them apart much more difficult.
The version control wrangling could end up taking longer than the change itself.
3. Managing temporary environment changes
// XXX: Changed the number here for faster testing. Change it back.
Our code has an event-triggering threshold: when event A happens enough times, it triggers event B. On this branch, we sometimes want to manually trigger event B after only a few occurrences of A, so we've temporarily lowered this threshold. But we don't want to accidentally push that change out, so we added an XXX comment.
Because we're working on a very stable part of the system, it doesn't make sense to lower the threshold permanently for our development environment. Changing the threshold based on the environment adds application complexity. Worse, it's yet another way that dev diverges from production, which is never desirable. We decided that it was better for this branch to diverge temporarily than for the dev environment to diverge permanently.
An XXX comment allows us to keep using the lower threshold, and gives us peace of mind that we won't accidentally push this dangerous change to production.
4. Documenting our work
The application-specific details are removed from this example. But its structure is as we wrote it in the real code.
/* XXX: Write summary comment explaining this module
* - Mention that the foo() function is responsible for frobbing.
* - Mention that bar() hits Stripe, so avoid multiple calls to it. But none
* of the other functions is allowed to hit Stripe, so call them at will.
*/
This branch introduces a major new module to the system. The module will need a summary comment, so it's tempting to write it right away. But writing module-level comments early on can be risky.
First, we may end up reverting our current changes, and this module may not even exist when the branch lands. If we write a perfect, nicely-formed comment, that effort might be wasted.
Second, this module or the other modules around it might change during the lifetime of our branch. We'll have a better sense for how the module fits into the system once the branch is stabilized, so that's a better time to finalize a high-level comment.
Finally, even if neither of those apply, the usual reasoning does still apply: we don't want to interrupt the change that we're making right now.
Even if we don't write the big comment up front, we can still make notes about what to include in it. Using a bulleted list lets us capture notes in the moment, without needing to take the time to write a full, nicely-formatted comment. This is a great example of how using the code as a to-do list balances tracking future work vs. interrupting our current task.
ESLint in action
Once the comments are in place, we can see them in our ESLint output. Our dev script automatically runs ESLint on every source file change, so we'll see this continually. This is also the output that will show up in CI, blocking deploys to production.
(The filenames are anonymized here because the exact details of the new subsystem aren't important.)
[eslint:server] src/server/old-subsystem/old-file.ts
[eslint:server] 63:13 warning Unexpected 'xxx' comment: 'XXX: This is inserting "Start Case"...' no-warning-comments
[eslint:server]
[eslint:server] src/server/new-subsystem/types.d.ts
[eslint:server] 5:1 warning Unexpected 'xxx' comment: 'XXX: Remove this module; it's trivial...' no-warning-comments
[eslint:server]
[eslint:server] src/server/new-subsystem/index.ts
[eslint:server] 7:1 warning Unexpected 'xxx' comment: 'XXX: Write summary comment explaining...' no-warning-comments
[eslint:server] 94:11 warning Unexpected 'xxx' comment: 'XXX: Changed the number here for faster...' no-warning-comments
[eslint:server]
[eslint:server] ✖ 4 problems (0 errors, 4 warnings)
[eslint:server]
[eslint:server] ESLint found too many warnings (maximum: 0).
Development flow
In large changes, it's common for us to have dozens of these XXX lint errors live at once. Usually, the arc of a large change is:
By the time we're resolving XXX comments, some will be outdated due to other changes made after we added the comment. That's fine; it's just a comment, and we can delete it. (But if we'd made the change at the time, that work would've been wasted!)
Even when the comments are still current, they're often quick changes that were just a bit too big to distract ourselves with at the time. We estimate that the downward slope from "peak XXX" to "0 XXX" is usually around 10% of the total development time for a branch. All of that work was going to happen one way or another; we just shifted it in time to aid our concentration and to allow better risk prioritization.
Using the linter to block XXXs from going to production is a great alternative to keeping a separate to-do list, or writing a test for every change we think of. Writing XXX comments isn't just faster; it protects our focus and prioritizes our attention. Give this approach a try on your next project!
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [1] => Array ( [uri] => https://www.executeprogram.com/blog/typescript-features-to-avoid [title] => TypeScript Features to Avoid [timestamp] => 1642542204 [author] => [content] =>This post lists four TypeScript features that we recommend you avoid. Depending on your circumstances, there may be good reasons to use them, but we think that avoiding them is a good default.
TypeScript is a complex language that has evolved significantly over time. Early in its development, the team added features that were incompatible with JavaScript. Recent development is more conservative, maintaining stricter compatibility with JavaScript features.
As with any mature language, we have to make difficult decisions about which TypeScript features to use and which to avoid. We experienced these trade-offs firsthand while building Execute Program's backend and frontend in TypeScript, and while creating our comprehensive TypeScript courses. Based on our experience, here are four recommendations about which features to avoid.
I. Avoid enums (See our lesson)
Enums give names to a set of constants.
In the example below, HttpMethod.Get is a name for the string 'GET'.
The HttpMethod type is conceptually similar to a union type between literal types, like 'GET' | 'POST'.
enum HttpMethod {
Get = 'GET',
Post = 'POST',
}
const method: HttpMethod = HttpMethod.Post;
method; // Evaluates to 'POST'
Here's the argument in favor of enums:
Suppose that we eventually need to replace the string 'POST' above with 'post'.
We change the enum's value to 'post' and we're done!
Other code in the system only references the enum member via HttpMethod.Post, and that enum member still exists.
Now imagine the same change with a union type instead of an enum.
We define the union 'GET' | 'POST', then later we decide to change it to 'get' | 'post'.
Any code that tries to use 'GET' or 'POST' as an HttpMethod is now a type error.
We have to update all of that code manually, which is an extra step when compared to enums.
This code maintenance argument for enums isn't very strong. When we add a new member to an enum or union, it rarely changes after creation. If we use unions, it's true that we may have to spend some time updating them in multiple places, but it's not a big problem because it happens rarely. Even when it does happen, the type errors can show us which updates to make.
The downside to enums comes from how they fit into the TypeScript language. TypeScript is supposed to be JavaScript, but with static type features added. If we remove all of the types from TypeScript code, what's left should be valid JavaScript code. The formal word used in the TypeScript documentation is "type-level extension": most TypeScript features are type-level extensions to JavaScript, and they don't affect the code's runtime behavior.
Here's a concrete example of type-level extension. We write this TypeScript code:
function add(x: number, y: number): number {
return x + y;
}
add(1, 2); // Evaluates to 3
The compiler checks the code's types.
Then it needs to generate JavaScript code.
Fortunately, that step is easy: the compiler simply removes all of the type annotations.
In this case, that means removing the : numbers.
What's left is perfectly legal JavaScript code.
function add(x, y) {
return x + y;
}
add(1, 2); // Evaluates to 3
Most TypeScript features work in this way, following the type-level extension rule. To get JavaScript code, the compiler simply removes the type annotations.
Unfortunately, enums break this rule.
HttpMethod and HttpMethod.Post were parts of a type, so they should be removed when TypeScript generates JavaScript code.
However, if the compiler simply removes the enum types from our code examples above, we're still left with JavaScript code that references HttpMethod.Post.
That will error during execution: we can't reference HttpMethod.Post if the compiler deleted it!
/* This is compiled JavaScript code referencing a TypeScript enum. But if the
* TypeScript compiler simply removes the enum, then there's nothing to
* reference!
*
* This code fails at runtime:
* Uncaught ReferenceError: HttpMethod is not defined */
const method = HttpMethod.Post;
TypeScript's solution in this case is to break its own rule. When compiling an enum, the compiler emits extra JavaScript code that never existed in the original TypeScript code. Few TypeScript features work like this, and each adds a confusing complication to the otherwise simple TypeScript compiler model. For these reasons, we recommend avoiding enums and using unions instead.
Why does the type-level extension rule matter?
Let's consider how the rule interacts with the ecosystem of JavaScript and TypeScript tools. TypeScript projects are inherently JavaScript projects, so they often use JavaScript build tools like Babel and webpack. These tools were designed for JavaScript, and it's still their primary focus today. Each tool is also an ecosystem of its own. There's a seemingly-endless universe of Babel and webpack plugins to process code.
How can Babel, webpack, their many plugins, and all of the other tools and plugins in the ecosystem fully support TypeScript? For most of the TypeScript language, the type-level extension rule makes these tools' jobs relatively easy. The tools strip out the type annotations, leaving valid JavaScript.
When it comes to enums (and namespaces, which we'll see in a moment) things are more difficult.
It's not good enough to simply remove enums.
The tools have to turn enum HttpMethod { ... } into working JavaScript code, even though JavaScript doesn't have enums at all.
This brings us to the practical problem with TypeScript's violations of its own type-level extension rule. Tools like Babel, webpack, and their plugins are all designed for JavaScript first, so TypeScript support is just one of their many features. Sometimes, TypeScript support doesn't receive as much attention as JavaScript support, which can lead to bugs.
The vast majority of tools will do a good job with variable declarations, function definitions, etc.; all of those are relatively easy to work with. But sometimes mistakes creep in with enums and namespaces, because they require more than just stripping off the type annotations. You can trust the TypeScript compiler itself to compile those features correctly, but some rarely-used tools in the ecosystem may make mistakes.
When your compiler, bundler, minifier, linter, code formatter, etc. silently miscompiles or misinterprets an out-of-the-way piece of your system, it can be very difficult to debug. Compiler bugs are notoriously difficult to track down. Note these words: "over the week, with the help of my colleagues, we managed to get a better understanding of the scope of the bug." (Emphasis added.)
II. Avoid namespaces (See our lesson)
Namespaces are like modules, except that more than one namespace can live in a single file. For example, we might have a file that defines separate namespaces for its exported code and for its tests. (We don't recommend doing this, but it's a simple way to show off namespaces.)
namespace Util {
export function wordCount(s: string) {
return s.split(/\b\w+\b/g).length - 1;
}
}
namespace Tests {
export function testWordCount() {
if (Util.wordCount('hello there') !== 2) {
throw new Error("Expected word count for 'hello there' to be 2");
}
}
}
Tests.testWordCount();
Namespaces can cause problems in practice. In the section on enums above, we saw TypeScript's "type-level extension" rule. Normally, the compiler removes all of the type annotations, and what's left is valid JavaScript code.
Namespaces break the type-level extension rule in the same way as enums.
In namespace Util { export function wordCount ... }, we can't remove the type definitions.
The entire namespace is a TypeScript-specific type definition!
What would happen to the other code outside of the namespace calling Util.wordCount(...)?
If we delete the Util namespace before generating JavaScript code, then Util doesn't exist any more, so the Util.wordCount(...) function call can't possibly work.
As with enums, the TypeScript compiler can't simply delete the namespace definitions. Instead, it has to generate new JavaScript code that doesn't exist in the original TypeScript code.
For enums, our suggestion was to use unions instead. For namespaces, we recommend using regular modules. It may be a bit annoying to create many small files, but modules have the same fundamental functionality as namespaces without the potential downsides.
III. Avoid decorators (for now)
Decorators are functions that modify or replace other functions (or classes). Here's a decorator example taken from the official docs.
// This is the decorator.
@sealed
class BugReport {
type = "report";
title: string;
constructor(t: string) {
this.title = t;
}
}
The @sealed decorator above alludes to the C# sealed modifier, which prevents other classes from inheriting from the sealed class.
We'd implement it by writing a sealed function that takes a class and modifies it to prevent inheritance.
Decorators were added to TypeScript first, before beginning their standardization process in JavaScript (ECMAScript). As of January 2022, decorators are still a stage 2 ECMAScript proposal. Stage 2 is "draft". The decorator proposal also seems to be stuck in committee purgatory: it's been at stage 2 since February of 2019.
We recommend avoiding decorators until they're at least a stage 3 ("candidate") proposal, or stage 4 ("finished") for more conservative teams.
There's always a chance that ECMAScript decorators will never be finalized. If that happens, they'll end up in a similar situation to TypeScript enums and namespaces. They'll continue to break TypeScript's type-level extension rule forever, and they'll be more likely to break when using build tools other than the official TypeScript compiler. We don't know whether that will happen or not, but the benefits of decorators are minor enough that we'd rather wait and see.
Some open source libraries, most notably TypeORM, use decorators heavily. We recognize that following our recommendation here precludes using TypeORM. Using TypeORM and its decorators is a fine choice, but it should be done intentionally, recognizing that decorators are currently in standardization purgatory and may never be finalized.
IV. Avoid the private keyword (See our lesson)
TypeScript has two ways to make class fields private.
There's the old private keyword, which is specific to TypeScript.
Then there's the new #somePrivateField syntax, which is taken from JavaScript.
Here's an example showing each of them:
class MyClass {
private field1: string;
#field2: string;
...
}
We recommend the new #somePrivateField syntax for a straightforward reason: these two features are roughly equivalent.
We'd like to maintain feature parity with JavaScript unless there's a compelling reason not to.
To recap our four recommendations:
#somePrivateField over private somePrivateField.Even when avoiding these features, it's good to have a working knowledge of them. They show up often in legacy code, and even in some new code. Not everyone agrees that they should be avoided. Execute Program's TypeScript courses teach these features for the reasons that we explained in our post on Teaching the Unfortunate Parts.
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [2] => Array ( [uri] => https://www.executeprogram.com/blog/teaching-the-unfortunate-parts [title] => Teaching the Unfortunate Parts [timestamp] => 1604958204 [author] => [content] =>When explaining a technology, we have to decide how to approach its shortcomings. There might be mistakes in its design, or it might have usability problems, or it might be unreliable. How do we approach these and how much emphasis do we place on them?
One approach is: "This tool has a lot of problems, but we'll show you how to avoid them." That can demotivate the learner: "Why am I learning this thing if it has so many problems?"
Another common approach is to never call out the problems at all. That can also demotivate the learner, but the demotivation is delayed. When they finish your video, book, etc. and use the tool in practice, they suddenly encounter the sharp edges. Now they might think "I haven't heard about these problems so this must be my fault."
In Execute Program, we try to avoid both of the above approaches. Instead, we aim for neutral descriptions in our courses. Where problems exist, we describe them flatly and directly. Often, probably due to personal writing quirks, our descriptions involve the word "unfortunate".
Here are four times when we used the word "unfortunate" or "unfortunately" in a lesson. Each is expanded here because the original surrounding lesson context is missing, but the original "unfortunate" is retained in each example.
1. NaN
(From the Modern JavaScript / isNaN lesson. The lesson links in this post, including that one, will take you directly to the interactive lesson, which doesn't require registering an account.)
Any operation on a NaN returns another NaN.
Unfortunately, that means that NaNs will propagate through the system.
By the time you actually see the NaN, it might have ended up in a very different place from where it started.
> const lastIndex = ['a', 'b', 'c'].elngth - 1;
const middleIndex = Math.floor(lastIndex / 2);
middleIndex;
NaN
Note the typo: elngth.
That returns undefined, then we subtract 1 from it giving NaN, then we divide by 2, then we call Math.floor.
Now imagine that each of those steps happened in a different function.
All we know is that a NaN came out the end.
Now we have to ask the perpetual question in dynamic languages: which function (or class, or module) caused the NaN (or null, undefined, nil, None, etc.), and which functions merely propagated it?
NaN itself is part of the IEEE 754 floating point standard.
It behaves the way it does for a reason.
But this is still a JavaScript problem because JavaScript returns NaN in so many cases where other languages don't.
In our example above, the NaN showed up when we did undefined - 1.
Other dynamic languages (like Ruby or Python) throw an error when we try to do that, and most static languages prohibit it at compile time.
This is a problem specific to JavaScript.
2. Mapping over maps
(From the Modern JavaScript / Maps lesson.)
Modern versions of JavaScript support a map data type. It's kind of like an object where the keys can be any type, not just strings. JavaScript also has a map method on arrays, which transforms the array's values into other values.
The data type Map and the method map are related at the conceptual level: they both "map" (or "relate") things to other things.
Other than that, they have no relationship.
Their identical names are just an unfortunate accident of history.
3. No negative array indexes in JavaScript
(From the JavaScript Arrays / Negative array indexes lesson.)
Ruby and Python both allow negative indexing on arrays.
In those languages, arr[-1] means "the last element of the array".
JavaScript has a concept of negative indexes: we can call arr.slice(-1) to get an array containing only the last element of arr.
We might expect that to work for normal array access as well, as it does in Ruby and Python: someArray[-1].
Alternately, we might expect someArray[-1] to throw an exception.
Unfortunately, it doesn't do either of those.
Instead, it returns undefined, the same value that we get when accessing any other index that doesn't exist.
> const arr = ['a', 'b', 'c'];
arr[-1];
undefined
4. TypeScript's unsoundness
(From the TypeScript / Type soundness lesson.)
A sound type system is one that fully enforces its own rules. TypeScript is unsound, which means that the compiler will sometimes accept code that violates the compiler's own rules.
In the example below, we create an array of strings. "Array of strings" means that it should never contain anything else. It certainly shouldn't contain any numbers.
Then we assign the array to a new variable with the type string | number.
That lets us push a number into the array.
We've now violated the original array variable's type: it says it's a string[], but it contains both strings and numbers.
> let names: string[] = ['Amir', 'Betty'];
let unsoundNames: (string | number)[] = names;
// This is unsound! It causes the "names" array to contain a number!
unsoundNames.push(5);
names;
['Amir', 'Betty', 5]
This is very bad, and unfortunately there's no way to fix it.
Being honest about problems
It's important to acknowledge design problems, especially when teaching. Learners are new to the topic, so they rarely have enough context to analyze designs critically. For example, someone learning TypeScript as their first static language won't know what soundness is.
People are going to hit TypeScript's unsoundness in real code.
They'll find NaNs propagating through their JavaScript systems running in production.
They'll wonder why the map data type and the map method have the same name even though they don't interact.
Their first thought will be that they're just missing something.
As authors, we could try to head this off with "OK, this technology has a ton of problems... in fact, it's pretty bad, but here we go, let's learn it!" That sets the learner up for demotivation from the start. Alternately, we could describe only the good parts of the tool, staying silent about the sharp edges. But that sets the learner up for demotivation later when they encounter sharp edges and blame themselves.
It's better to describe the good parts, then tell the learner that, unfortunately, TypeScript's type system is also unsound. But it's unsound for a reason: it's a trade-off that its designers made to achieve compatibility with the existing JavaScript ecosystem. Now the learner knows that the problem is real and they're not imagining it; they know that the problem isn't their fault; and, if the lesson did its job, they know how to spot the problem and work around it.
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [3] => Array ( [uri] => https://www.executeprogram.com/blog/typescripts-excess-properties-can-bite-you [title] => TypeScript's excess properties can bite you [timestamp] => 1593117804 [author] => [content] =>Recently, our Cypress tests started hanging silently. This post explains the cause: because of a subtle part of TypeScript's language design, some circular data structures were invisible in the code, which caused a serialization error, which Cypress handled incorrectly. If we don't understand the subtleties of our tools, bugs like this can be inscrutable; but if we learn the subtleties, they can be quick, if annoying, speed bumps.
1. Excess object properties
TypeScript's type system is unusual in many ways. Here's an example highlighting one of them:
type User = {
name: string,
}
function buildAmir() {
return {
name: 'Amir',
age: 36,
}
}
const amir: User = buildAmir()
console.log(amir)
In most statically typed languages, this code wouldn't compile, but TypeScript allows it.
What comes out when we run it?
Amir has the type {name: string}, so it seems like the answer should be {name: 'Amir'}.
What actually comes out at runtime is:
{name: 'Amir', age: 36}
This arises from two design decisions in TypeScript.
First, objects are allowed to have excess keys and still satisfy an object type that doesn't have those keys.
The excess keys are ignored by the type system.
For example, the compiler wouldn't let us say amir.age, even though the age property does exist at runtime!
Second, the TypeScript compiler emits code by stripping the types away. What's left is JavaScript. For our example, the compiler emits JavaScript roughly like this:
function buildAmir() {
return {
name: 'Amir',
age: 36,
}
}
const amir = buildAmir()
console.log(amir)
When we look at it that way... of course the output is { name: 'Amir', age: 36 }.
That's what the code says!
At first glance, this all seems fine. So how did it make our tests silently fail to run?
2. Large objects
Internally, Execute Program has an Example type representing a single interactive code example in a lesson.
Its type looks like this, heavily simplified and tweaked to make sense in isolation.
type Example = {
kind: "example"
id: string
code: string
expectedResult: string
isInteractive: boolean
}
For example, we might have an example where code is '1 + 1' and expectedResult is '2'.
In that case, Execute Program would show the user 1 + 1 and wait for them to type 2.
In the past, the lessons referenced their examples, but the examples didn't know about the lessons that contain them.
(There's no lesson property in the object above.)
That was done because backreferences would create a reference cycle: the lesson references the example, which references the lesson, which references the example, and so on forever.
Reference cycles make garbage collectors work harder, as well as causing confusing bugs like the one that we're working toward here.
(We'll use "reference cycle" and "circular data" interchangeably in this post.)
It's inconvenient for examples not to know about their lessons, so we recently added some lesson data to the example type.
But not a full lesson, just a little bit of data: the containing lesson's ID.
This made a particular new feature much easier to implement.
Now the Example type becomes:
type Example = {
// This stuff is all the same as before.
kind: "example"
id: string
code: string
expectedResult: string
isInteractive: boolean
// This is new.
lesson: {
id: string
}
}
No reference cycles!
That little lesson object only has an id, not a list of examples that would cause a cycle.
But unfortunately, we naively created the example object like this:
const example: Example = {
kind: "example",
id,
code,
expectedResult,
isInteractive,
lesson,
}
The lesson there is a full lesson object.
From TypeScript's perspective, example.lesson's type is {id: string}.
But at runtime it will be the entire lesson object, which is (1) huge, and (2) contains references to all of the lesson's examples, which contain references back to the lesson, which contains references to all of the examples, and so on.
The static types show no reference cycle, but one still exists.
3. Test data generation
Our Cypress tests need to load up our actual lessons for testing. Unfortunately, there's not a great way to do that in Cypress. What we do today is:
import that JSON file to access the lesson data.This seems clunky because it is clunky. But Cypress comes with a lot of constraints, and this was how we satisfied them.
When we added lesson to the Example object, the clunks came home to roost.
Trying to call JSON.stringify on any circular data structure will cause an exception in Node.
Here's a shell command demonstrating:
$ node -e 'const x = {}; x.x = x; JSON.stringify(x)'
TypeError: Converting circular structure to JSON
The bug
The JSON.stringify call on our curriculum object definitely threw an exception inside our Cypress plugin code.
But the way that manifested in Cypress was... nothing.
Cypress launched the browser process to run the tests and it sat there, blank.
Having seen this movie before, I took a guess and opened Activity Monitor. As expected, the Cypress process was using 100% of the CPU. Eventually, CPU usage dropped to zero, but Cypress still showed nothing. I knew that errors in the plugin loading stage sometimes (maybe always?) pass silently, and I knew that we only had one plugin, and I knew that it was serializing a lot of data.
A quick check and... yes, every Example in the system contains a circular reference to its Lesson.
But then why did this cause 100% CPU usage for several seconds?
Shouldn't Node have errored immediately after seeing the circular reference?
What probably happened is: the curriculum object was so large that it took multiple seconds to get to the point where it could even check for cycles!
(As I write this, our course list contains 4,024 code examples, plus all of the accompanying text, all stored in fine-grained data structures with a lot of metadata.)
The fix
The bug is weird, but the fix is very simple. Here's the buggy code that creates a circular reference:
const example: Example = {
kind: "example",
id,
code,
expectedResult,
isInteractive,
lesson,
}
We need to avoid putting the entire lesson object inside the example.
Instead, we pull out only the property we care about, the ID:
const example: Example = {
kind: "example",
id,
code,
expectedResult,
isInteractive,
lesson: {
id: lesson.id
},
}
Now there's no cycle and everything works again. A trivial solution to a complex problem implicating several subtle topics. To summarize:
Where do we place blame here?
TypeScript's design decisions, including this one, allow it to be flexible enough to interact with existing code on the web.
It's hard to argue against that; it's a major reason that we use TypeScript in the first place.
And JSON.stringify is correct to error on cyclic data because that data is unrepresentable in JSON.
Cypress' silent error handling, on the other hand, was a genuine bug. But suppose that it had surfaced the error, "TypeError: Converting circular structure to JSON". We'd still have to understand circular data structures, and we'd still have to understand the TypeScript design quirk that lets them show up even when the types don't allow them.
That leaves us with this plan for debugging problems like this:
This is a story about how we used io-ts to save React useReducer state for later.
Saved state may be restored months later, in a different browser, while both the React code and the data used by the reducer have continued to change over time.
The solution described here is in production on the site you're looking at.
First, a short description of the application itself:
Execute Program is an interactive learning platform made up of courses (like TypeScript or Regular Expressions). Each course is made up of many lessons (like Generic function types or Character classes). And each lesson is made up of interactive code examples mixed with paragraphs of text.
As a user goes through a lesson, they reveal one example or paragraph after another. We call those "steps": each code example is a step and each paragraph is a step. When they reach the last step, the client code tells the server that the user has finished the lesson.
Here's the problem:
Here's a video of our solution in action. We advance through a paragraph and an initial, non-interactive code example. Then we reload the page and choose to resume the lesson. Then we reload the page again and choose to restart the lesson.
Let's examine a series of possible solutions, beginning with the simplest.
Solution 1: Store the current step index.
React's useReducer manages large state objects with complex transitions.
The developer defines how the object will change in response to user actions.
For example, when we clicked the "Continue" button in the example above, our reducer incremented the step index by 1.
In our lessons, the reducer state contains the full step objects as retrieved from the API, including the actual code and paragraph text. It also contains various metadata, including an index variable: a number representing the current step's index in the step array. When the user finishes a step – for example, by completing a code example – we increment the index and the UI shows the next step.
The first possible solution is to save the step index somewhere: maybe on the server, or maybe in the browser's localStorage.
This works... but only if our lessons never change.
Our lessons changes almost every day as we tweak them in response to user feedback. If we insert, delete, or reorder steps, then the stored index will be wrong. The user will get to a certain point in a lesson, then leave, then come back, and the lesson will resume at the wrong step. We need a solution that always resumes to the correct point in the lesson.
Solution 2: Invalidate the step index when the lesson changes.
We can store the step index on the server or in the browser's local storage, but also have a way to invalidate it when the lesson content changes. The easiest solution here is a hash over the entire lesson's content: all of the code examples and all of the explanatory text. We store the hash alongside the step index.
Later, when a user returns to the lesson, we compare the stored hash against the current lesson hash. If they match, we offer to resume the lesson. Any content change will result in a different hash, so we'll ignore the stored step index whenever the lesson changes. This works... but it only restores the step index. What about the other metadata that the client gathers during the lesson?
We care about that metadata! For example, we want to know how many attempts the user made at each code example. Aggregating those numbers can show us when certain examples are too difficult.
We need a solution that preserves that metadata in addition to the step index. One idea is to pull out the pieces of the reducer state that matter and store only them. That would include the exact content of the lesson (or a hash of it), the number of attempts the user made at each code example, etc.
Solution 3: Store the entire reducer state.
As it turns out, we care about most of the data in the reducer state; otherwise we wouldn't be tracking it! The easiest solution is to store the entire reducer state, then restore it later when the user resumes the lesson.
Storing the reducer state brings up another data mismatch problem. Last time, the mismatch was in the lesson content. Now, the mismatch is in the schema or structure of the reducer's state object: the actual keys in the state, and the particular types of data in those keys.
Here's the problem that will occur:
We need a solution that can ignore any saved reducer state that was created by an older, incompatible version of the client-side code.
Solution 4: Version the reducer state.
The obvious solution is to add a version number to the saved state data. When we make an incompatible change to the reducer state's structure, we change the version number. When restoring an old reducer state, we compare its version number against the current code's version number. If they don't match, we ignore the saved state and start from the beginning of the lesson.
Unfortunately, humans are bad at remembering to do things. It's likely that we'd change the reducer state's structure at some point, but forget to change the version number. This is especially likely because the state contains the lesson content itself, which comes from our API. We'd have to remember to increment the version number when changing the state, but also whenever we change any of the related API endpoints. That's too error-prone.
We could automate the version number. For example, we could hash the source files that define the relevant parts of the API, as well as the file that defines the reducer state, and use that hash as the "version number". But that has its own problems.
First, even trivial changes like adding or removing whitespace would invalidate all lesson states for all users. Second, what happens if we break one of those API files into two different files? If we forget to add those two new files to our hashing scheme, then the system won't know to invalidate saved states when they change. Hashing the source files would only provide the illusion of safety; in reality, it's probably more error-prone than a manual version number.
We need a solution that can detect changes to the structure of the state data only when that structure genuinely changes, and regardless of why it changed.
Interlude: An introduction to io-ts.
Execute Program is written in TypeScript. It already uses io-ts to do runtime validation of API data. For example, when an API request comes in, we use io-ts to check that all of the keys are present; that keys that should be numbers are numbers; that arrays are arrays; etc.
Our reducer problem is similar to the API problem. In an API, we have network data coming in from another computer. It could be coming from a malicious attacker, or it could be coming from a future version of ourselves who made a mistake and sent the wrong data to our own API. We want to detect those problems and throw an error.
In our reducer state problem, we also have data coming in from "another" computer. In this case, that computer is the user's own computer from some point in the past: the state was saved a month ago and it's being resumed now. Like with the API, we need to check the structure of the data: make sure that all expected keys are present; that keys that should be numbers are numbers; etc. We can use io-ts to solve this, just as we do with our API.
Here's a quick primer on how io-ts works.
Suppose that our API has a register endpoint for registering new accounts.
It accepts an email address.
We can define an io-ts codec that validates the incoming data:
import * as t from 'io-ts'
import { isRight } from 'fp-ts/lib/Either'
const register = t.strict({
email: t.string,
})
Then, we can use it to check for whether some data matches our codec:
// This returns true.
isRight(register.decode(
{email: 'amir@example.com'}
))
// Each of these returns false.
isRight(register.decode(
{email: 1234}
))
isRight(register.decode(
{email: undefined}
))
isRight(register.decode(
{userEmail: 'amir@example.com'}
))
isRight(register.decode(
{}
))
isRight(register.decode(
null
))
This is how Execute Program checks API data.
We define io-ts codecs for every API endpoint.
Then the server uses decode and isRight to validate all incoming and outgoing data.
Some of our API data is complex: objects nested in arrays nested in other objects nested in yet other objects, etc. We break those complex io-ts codecs into smaller, more manageable pieces, which are sometimes shared between multiple endpoints.
There's one more important fact about io-ts.
We're in TypeScript, so we want static types for our data.
For our register codec above, that might look like:
type Register = {
email: string
}
But that duplicates our io-ts codec. Instead of writing the type manually, we can extract the exact type above from the io-ts validator:
type Register = t.TypeOf<typeof register>
That will look quite arcane if you're new to TypeScript.
But this pattern always looks the same when extracting a static type from an io-ts codec, so it becomes routine with with some TypeScript and io-ts experience.
One short line of code using t.TypeOf can replace hundreds of lines of duplicated type definitions.
(Full disclosure: doing this sometimes makes type error messages more confusing, but the details of that are out of scope here. In some situations, we do define the types explicitly, effectively duplicating the io-ts codec definitions. It's annoying, but the type system will keep us honest: it will error if our types don't match the codec's structure. These definitions are used all over the system but change rarely, so a bit of extra code in a complex codec is often a good trade-off to get better error messages.)
To summarize:
Solution 5: Validate the reducer state with io-ts.
Now we can talk about our chosen solution for the lesson resuming problem.
We define an io-ts codec that describes the structure of the reducer state. Remember that the reducer state contains the steps (the code examples and paragraphs), which come from the API. Our entire API is defined using io-ts, so we can reuse all of those codecs. Excluding imports and whitespace, the io-ts codecs for our reducer state are 66 lines long.
If we ever change the reducer state's structure, we'll be forced to update the io-ts codec. That's because the TypeScript type system checks all of this: the io-ts codec must match the reducer state types, and the reducer code in our React components must also match those types.
Now we can formulate a solution to the problem of deciding whether a lesson state came from an old, incompatible version of the application:
Now we have to solve the second problem: what if the lesson content has changed? Fortunately, the reducer state contains all of the steps (the code examples and paragraphs) as they existed when the state was saved. The client also has the current examples and paragraphs: it's showing a lesson, so it just got all of the lesson content from the API. We can compare the two to decide whether the content has changed since this state was saved:
isEqual to do a deep comparison of the saved state's steps and the current API steps.This is a full solution to the problem! The type system ensures that we never create invalid reducer states in our normal React code. The io-ts codecs ensure that we never restore an old state with the wrong structure. And checking the saved steps against the current steps ensures that we don't confuse the user or the app by restoring to the wrong point.
Here's the function that loads the state. It takes a string that (hopefully) contains a JSON-serialized state.
export function loadSerializedState(
apiSteps: readonly ApiStep[],
serializedModelState: string
): State | undefined {
return validateSteps(
apiSteps,
validateStateType(
deserializeJson(serializedModelState)
)
)
}
The three functions called here are quite short. Excluding imports and comments, all four functions sum to 38 lines of code.
Each of the three functions called here will return undefined if something doesn't match, and each of them passes undefineds along.
That allows us to write the top-level loadSerializedState function in this simple way.
If loadSerializedState returns an object, then we know it's a valid reducer state that corresponds to the current lesson content and the current client-side code.
We feed it into useReducer, and React uses it without knowing or caring that it was created a month ago!
It's important to note that this is a conservative solution. Suppose that we make a typo in the lesson content: we write "proimse" instead of "promise". When we fix that typo, the current lesson content will no longer match any of the saved states for that lesson. Every saved state for that lesson is immediately invalidated.
In the future, we may try to detect trivial content changes and allow those saved states to be restored. But the perfect is the enemy of the good, and we get most of the benefit without doing any deep content comparison.
Where do we store the state?
Now that we can save and restore our reducer state, there's one more question: where do we store it? The initial plan was to store it in the browser's local storage. But that has a big problem. If a user begins a lesson on their phone, then later tries to resume it on their laptop, the state won't be there. The laptop's browser can't see the saved state on the phone.
Our actual solution is to store the state in our server-side database. This feels wrong at first glance: we're storing a client-side React component's state as an opaque JSON blob in our server-side database, and we're doing that knowing that some of those JSON blobs will go out of date and be unusable in the future. But io-ts and the type system keep us honest here! And this is a convenience feature, so we can always change it later, even if that means throwing away all of the saved states. (When a user finishes a lesson, that record is stored in a separate part of the database.)
A simple solution
It took a lot of words to describe this solution, but that's because there was a lot of background to explain. In reality, I went almost directly to the io-ts solution because I could see that this problem was analogous to the API validation problem.
This code change was quite simple: it added 470 lines of code containing 1,102 words. (This article contains almost three times as many words!) Those 470 lines include the new modal UI for resuming, the new API endpoint, the API server handler, the database migration, and the database model class, all of which would exist with or without this solution based on io-ts.
Our tools are a good match for our problems, which allows us to solve problems like this smoothly:
useReducer represents states as simple JavaScript objects, so we can serialize them to JSON and bring them back later.If you're interested in trying Execute Program to see this in action, we recommend the TypeScript courses or the Regular Expressions course. Please use the feedback item in the menu to tell us what you think!
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [5] => Array ( [uri] => https://www.executeprogram.com/blog/problems-with-typescript-in-2020 [title] => Problems With TypeScript in 2020 [timestamp] => 1586983404 [author] => [content] =>TypeScript is very good. We ported our frontend and backend to it with no regrets. We have a much smaller test suite than we ever could've had in a dynamic language. All of this is great. Still, there are downsides to TypeScript, and we should be honest about them too.
JavaScript
TypeScript is a superset of JavaScript, and JavaScript is a mess. It was designed in 10 days, then it evolved haphazardly for almost two decades, then it was finally cleaned up in the last few years. A lot of the warts are still there and will always be there. For example:
1 * '1' returns 1 instead of an error.({name: 'Amir'}).age returns undefined rather than erroring.x && x + 1 looks like it means "add 1 to x if it's not null or undefined"; but when x is 0 it returns 0, not 1.TypeScript can't fix all of JavaScript's warts. But having used JavaScript since the 90s, I'm amazed at how well TypeScript did its dirty, improbable job. The first two code examples in the previous list are illegal in TypeScript. By adding a linter, the third one becomes illegal too. Those examples are simple, but TypeScript has saved us from many complex, subtle bugs that are too complex to show here.
The TypeScript team has been working to standardize new JavaScript features that let them increase safety further. For example, the new nullish coalescing operator mitigates JavaScript's design defects around the falsiness of 0 and the empty string, "".
Overall, the JavaScript underneath TypeScript is still frustrating in many situations, but TypeScript mitigates many of the worst frustrations. (A past version of myself would be very surprised to see this future!)
Compiler bugs
For decades, programmers joked that "It's never a compiler bug." Compilers weren't perfect, but "compiler bugs" almost always turned out to be the programmer misunderstanding the language.
When used as a long-lived watcher process, TypeScript has broken this trend: it's buggy. The compiler tends to hit bugs when files are deleted or renamed. We have a special script that notices those events and restarts the compiler. But our script is imperfect, so we still end up restarting the compiler manually. Last Saturday, I manually restarted the TypeScript compiler dozens of times as it encountered multiple bugs over and over again.
Sometimes the compiler doesn't realize that a file was deleted, so it produces incorrect errors. For example, if file A imports file B, and we delete both, it will sometimes complain that file A is trying to import B, which doesn't exist. But A doesn't exist either, so it shouldn't be causing an error! Other times, the compiler "successfully" compiles when there are actually type errors. In all of those cases, killing and restarting the compiler makes the errors go away, which is an unambiguous sign that it's a compiler bug in the watcher's state management.
(Webpack's watcher also has similar bugs around file deletion and renames. The easiest workaround for both TypeScript and Webpack is to restart all of the tools when a file is renamed or deleted, then wait as they all start back up. I feel embarrassed while waiting for those restarts.)
In the compiler's defense, this all seems to be limited to the watcher. I've never seen a bug in the compiler when it's cold booted, as it is during a production deploy or a CI run. And TypeScript doesn't seem worse than most other long-running development tools. As any long-time IDE user knows, any development tool that runs for a long time will break.
Unfortunately, booting the TypeScript compiler is very expensive, so we have no choice but to keep the compiler running between changes. This is in contrast to some other static languages like Go and Reason, which can compile systems the size of Execute Program in the neighborhood of 100 times faster than TypeScript. Those compilers are fast enough to be booted as-needed and then terminated. No need for a long-lived watcher.
The Node/NPM ecosystem
TypeScript is a superset of JavaScript, so it's closely connected to the Node and NPM ecosystems. Like any large ecosystem, the Node and NPM ecosystem is a mixed bag. There are so many packages; it's great! But some of them are very good and others are less good.
TypeScript allows us to write types for existing JavaScript libraries. People (often not a package's original authors) publish these type definitions in a central repository. On GitHub, that repository says "The repository for high-quality TypeScript type definitions." There's a bit of aspiration in that description. Sometimes it's true and the type definitions are high-quality! Sometimes they're not.
As one example, we recently did a major version upgrade of the database library that underpins our own database library. That made some of our valid code fail to type check. Worse, it allowed other code to type check even though it had major type errors. From our perspective, the "upgrade" was a step backward. But we don't have a choice if we want to continue to get security updates, and continue to upgrade other packages that depend on the database library.
None of that happened because TypeScript is bad; it happened because the type definitions for that library are wrong. Fortunately, our recent experience with that database library is an outlier. Most of the time, everything goes fine. Still, this is an annoyance that's relatively unique to TypeScript because it's a type system layered over an existing dynamic language.
It's hard to say whether the Node+NPM+TypeScript ecosystem is "better" or "worse" than, say, Ruby's or Python's. However, it's definitely far more complex, and it requires a more careful approach. Sometimes, there will be 50 packages that claim to do the thing that we need. Sometimes, one of them has more downloads than the other 49 combined. Sometimes, the best solution is the ninth-most-popular.
Overall, less-experienced programmers face an uphill battle in deciding which packages are worth using. Adding TypeScript increases that burden: now we have to analyze all of the library options, and we also have to analyze the presence and quality of third-party type definitions. Arguments for or against packages may be written by JavaScript users, so they won't be considering the quality of the library's TypeScript type definitions.
These problems don't outweigh the benefits. Compiler bugs are annoyances, but they've never affected our CI or production environments. Ecosystem complexity is a danger that can be navigated with forethought and by learning from past mistakes. I mention these problems to balance out the praise in earlier articles about TypeScript, but for us the trade-off is unambiguously in TypeScript's favor!
Gary Bernhardt
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [6] => Array ( [uri] => https://www.executeprogram.com/blog/are-tests-necessary-in-typescript [title] => Are Tests Necessary in TypeScript? [timestamp] => 1586810604 [author] => [content] =>We ported our React frontend from JavaScript to TypeScript, then did the same for our Ruby backend. It was a success; large classes of mistakes became impossible. You might wonder whether we could have achieved these results with full test coverage instead.
Type systems aren't good at everything. One common example is: type systems usually can't stop us from accidentally inverting a conditional. In most cases, switching the "if" and "else" won't cause a type error. For that, we have tests.
In Execute Program, we always write tests around subsystems that either (1) are critical to the product or (2) contain a lot of details that can't be statically checked. Billing is covered by tests because we don't want to charge someone incorrectly. Our progression course model is covered by tests because it contains a lot of conditional logic that the type system can't check.
We don't want tests covering most of our React components, though. That wouldn't help with the main difficulties in writing components:
It may still be a good idea to cover all of our components, even if our main component problems aren't solved by normal tests. We should do a cost/benefit analysis to find out what we'd be "paying" to get those tests.
First, what's our starting state? Our entire system is 24,065 lines including all current tests, with none of those tests covering the components directly. (Some of the components are partially exercised through browser-driving tests.)
Assuming a 2:1 test:code ratio for full coverage, covering our 9,000 lines of component code with unit tests would require 18,000 lines of test code. Adding 18,000 lines of client tests would almost double the total amount of code that we maintain. In practice, we already have a low defect rate in our TypeScript frontend, so there's no reason to pay that maintenance cost.
Although we don't test components directly, there are some other parts of the frontend code that are tested directly. For example, we have some high-value tests around some React reducers because they're tricky and full of conditionals. Those tests add up to around 300 lines, giving us a test:code ratio of about 1:30 for client code.
We also don't have tests covering our backend API handlers. Only a few of those handlers have any conditionals at all; they mostly translate API requests into calls to other subsystems. We push conditional logic down into those subsystems and test it there. As usual, TypeScript ensures that the handlers are properly wired up to the lower subsystems.
The subsystems below the API handlers do all kinds of different things. We have subsystems for tracking which lessons a user has unlocked; for computing statistics about their finished lessons; for managing subscriptions and billing; for deciding what content users have access to; etc. All of those subsystems are fully tested. That gives the server a roughly 1:2 test:code ratio: two lines of production code for every line of test.
All of this is to say: there are places where tests are necessary to achieve confidence in the system. But any production web application will have large regions where we can gain confidence via the type system and avoid writing almost all tests.
We have 19,498 lines of production code covered by 4,567 lines of tests. That gives an actual test:code ratio of about 1:5 overall. It feels about right.
Let's imagine an alternate version of Execute Program with a 2:1 test:code ratio. Naively covering all 19,498 lines of our production code with 2:1 tests would require 38,996 lines of test code, increasing our total line count from 24,065 lines to 58,494 lines. That's 243% as much code for very little additional benefit.
To answer the question that we started with: could we have achieved these results by forgetting TypeScript and fully covering the app with tests? The most literal answer is: yes, we might be able to get to the same low defect rate, but that ignores all other trade-offs in play here. We'd be maintaining an extra 150% or so more code, depending on our final test:code ratio. We'd also spend a lot more time writing and maintaining test code, rather than building features.
Types and tests are not equivalent at all; they give very different kinds of confidence. Tests can never provide an unbroken chain of checks, from the database all the way to the frontend React props, guaranteeing that all of the data has the right shape. Types can't (usually) tell us when we accidentally put a "!" in front of a conditional. Focusing on one or the other means sacrificing quality, work efficiency, or both.
Gary Bernhardt
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [7] => Array ( [uri] => https://www.executeprogram.com/blog/porting-to-typescript-solved-our-api-woes [title] => Porting to TypeScript Solved Our API Woes [timestamp] => 1586378604 [author] => [content] =>We ported our React frontend from JavaScript to TypeScript, but left the backend in Ruby. Eventually, we ported the backend to TypeScript too.
With the Ruby backend, we sometimes forgot that a particular API property held an array of strings, not a single string. Sometimes we changed a piece of the API that was referenced in multiple places but forgot to update one of those places. These are normal dynamic language problems in any system whose tests don't have 100% test coverage. (And it will still happen even with 100% coverage; it's just less likely.)
At the same time, these kinds of problems were gone from the frontend since porting it to TypeScript. I had more experience with backend development, but I was making more simple errors in the backend than the frontend. That was a sign that porting the backend was a good idea.
I ported the backend from Ruby to TypeScript in about two weeks in March of 2019. It went well! We pushed it to production, which was then in a closed beta, on April 14, 2019. Nothing blew up; users didn't notice. Here's a timeline of the run-up to the backend port, plus the time immediately after:
I wrote an unusual amount of custom infrastructure during that port. We have a custom 200-line test runner; a custom 120-line database model library; and a larger API router library that spans the frontend and backend code.
Of our custom infrastructure, the router is the most interesting piece to discuss. It wraps Express, enforcing the API types that are shared between the client and server code. That means that when one side of the API changes, the other side won't even compile until it's updated to match.
Here's the backend handler for the blog post list, one of the simplest in the system:
router.handleGet(api.blog, async () => {
return {
posts: blog.posts,
}
})
If we rename the posts key to blogPosts, we get a compile error ending with the line below.
(The actual object types are removed from the error message to keep it short here.)
Property 'posts' is missing in type '...' but required in type '...'.
Each endpoint is defined by an api.someNameHere object, which is shared between the client and server.
Notice that the handler definition doesn't name any types directly; they're all inferred from the api.blog argument.
This works for trivial endpoints like blog above, but it also works well for complex endpoints.
For example, our lesson API endpoint has a deep key .lesson.steps[index].isInteractive, which is a boolean.
All of these mistakes are now impossible:
isinteractive on the client or to return it from the server, that won't compile; it has to be isInteractive, with a capital "I".isInteractive, it won't compile.isInteractive in a variable of type number, it won't compile.isInteractive is a number rather than a boolean, then neither the client nor server will compile until they're fixed.None of that involves code generation; it's done using io-ts and a couple hundred lines of custom router code.
There's overhead in defining these API types, but it's not difficult. When changing the API's structure, we have to know how the structure is changing. We write our understanding down in the API type definitions, then the compiler shows us all of the places that have to be fixed.
It's difficult to appreciate how valuable this is until you've used it for a while. We can move large sub-objects in the API from one place to another, rename their keys, split one object into two separate objects, merge multiple objects into one new one, or split and merge entire endpoints, all without worrying about whether we missed a corresponding change in the client or server.
Here's a real example.
I recently spent around 20 hours redesigning Execute Program's API over four weekends.
The entire API structure changed, totaling tens of thousands of lines of diff across the API, server, and client.
I redesigned the server-side route definition code (like handleGet shown above); rewrote all of the type definitions for the API, making huge structural changes to many of them; and rewrote every part of the client that called the API.
246 of our 292 source files were modified in this change.
Throughout most of the redesign, I relied only on the type system. In the final hour of the 20-hour process, I started running the tests, which mostly passed. At the very end, we did a full manual testing pass and found three small bugs.
All three bugs were logic errors: conditionals that accidentally went the wrong way, which type systems usually can't detect. The bugs were fixed in a few minutes. That redesigned API was deployed a few months ago, so it served you this blog post (and everything else on Execute Program).
(This doesn't mean that a static type system will guarantee that our code is always correct, or guarantee that we don't need tests. But refactoring becomes much easier. We'll talk about the larger question of testing in the next post.)
There is one place where we do code generation: we use schemats to generate type definitions from our database structure. It connects to the Postgres database, looks at the columns' types, and dumps corresponding TypeScript type definitions to a normal ".d.ts" file used by the rest of the application.
The database schema type file is regenerated by our migration script on every migration run, so we don't do any manual maintenance on those types. The database models use the database type definitions to ensure that application code accesses every part of the database correctly. No missing tables; no missing columns; no putting a null in a non-nullable column; no forgetting to handle null in columns that are nullable; all statically verified at compile time.
All of that together creates an unbroken statically typed chain from the database all the way to the frontend React props:
Since finishing this port, I don't remember any API mismatch making it past the compiler. We've had no production failures due to mistakes where the two sides of the API disagree about the shape of the data. This isn't due to automated testing; we don't write any tests for the API itself.
These guarantees are wonderful: we can focus on the parts of the app that matter! I spend very little time wrangling types – far less than I spent chasing down confusing errors that propagated through layers of Ruby or JavaScript code before causing a confusing exception somewhere far away from the original source of the bug.
Here's a timeline of our development since the backend port. We've had a lot of time and new code to evaluate the results:
We haven't discussed a common objection to any post like this: couldn't you get the same result by writing tests? Absolutely not, which we'll cover in the next post!
Gary Bernhardt
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [8] => Array ( [uri] => https://www.executeprogram.com/blog/porting-a-react-frontend-to-typescript [title] => Porting a React Frontend to TypeScript [timestamp] => 1586205804 [author] => [content] =>The beta version of Execute Program was written in Ruby and JavaScript. Then we ported all of it to TypeScript in multiple steps. This is the story of the frontend port, which was the first step.
In Execute Program's original JavaScript frontend, I often made small mistakes. For example, I'd pass the wrong prop names to a React component, or leave a prop out, or pass the wrong kind of data. (Props are pieces of data sent as arguments to a React component. It's common for a component to pass some props to one of its sub-components, which pass them on again, etc.)
This is a big problem with dynamic languages like JavaScript and Ruby. I've been learning to mitigate it for 15 years. Here I am talking about it way back in 2011. Mitigations like the ones discussed there do help, but they don't scale well as a system grows, and there's no safety net for when we forget them.
I thought that 15 years was enough; I wanted to return to static type systems, where these mistakes are impossible. There were a few options: Elm, Reason, Flow, TypeScript, PureScript. (That's not an exhaustive list.) I decided on TypeScript because:
Porting the frontend JavaScript code to TypeScript took about 2 days in October of 2018. Here's a plot showing how much code we had in each language leading up to, and immediately after, that port.
That was pre-beta, so the system was still small: about 6,000 lines. The period after the port is missing here; we'll expand on that in future posts.
After that port, React prop problems disappeared. We'll look at a couple examples, starting with an easy one. Here's the code that renders the "Continue" button that appears after every paragraph of text in our courses:
<Button
autofocus={true}
icon="arrowRight"
onClick={continue}
primary
>
Continue
</Button>
The type of that Button component's props is shown below.
When reading a property type like autofocus?: boolean, "autofocus" is the name of the property; "?" means that it's optional; ":" separates the property name from its type; and "boolean" is the type.
The last property type, onClick, means "a function that takes no arguments and returns nothing".
(If TypeScript's function type syntax is unfamiliar, you can get a comprehensive overview in our lesson on TypeScript's function types.)
type ButtonProps = {
autofocus?: boolean
icon?: IconName
primary?: boolean
onClick: () => void
}
What happens if we change the "autofocus" prop from true to 1? We're now passing a number value where the type system expects a boolean. Less than a second later, the compiler prints the error below. (Some irrelevant details have been removed here; we'll do that for all of the errors in this series of articles.)
src/client/components/explanation.tsx(13,27):
error: Type 'number' is not assignable to type 'boolean | undefined'.
The offending code also turns red in vim. I fix it and the red goes away. Fixing the mistake takes seconds. In Ruby or JavaScript, I might spend minutes manually testing the app and rummaging around in its state to find out what happened. (I could also rely on automated tests, but we cover the issue of tests vs. types in another post.)
That integer-to-boolean change was a simple and low-stakes use of the type system.
Button's icon property shows more advanced use.
Here's the Button invocation again:
<Button
autofocus={true}
icon="arrowRight"
onClick={continue}
primary
>
Continue
</Button>
It looks like the icon prop is just a string: "arrowRight".
At runtime, in the compiled JavaScript code, it will be a string.
But in the ButtonProps type shown above, we defined it as an IconName, which is defined elsewhere.
Let's see what the type does before we look at its definition. Suppose that we change the "icon" prop to "banana". We don't actually have an icon named "banana".
<Button
autofocus={true}
icon="banana"
onClick={continue}
primary
>
Continue
</Button>
Less than a second later, the TypeScript compiler rejects that change:
src/client/components/explanation.tsx(13,44):
error: Type '"banana"' is not assignable to type
'"menu" | "arrowDown" | "arrowLeft" | ... 21 more ... | undefined'.
The compiler is saying that "icon" can't be any arbitrary string; it has to be one of the 24 strings that we've defined as icon names. The compiler will reject any change that leaves us referencing a non-existent icon; it's not a valid program and can't even begin executing.
There are multiple ways to implement the IconName type.
One is to write a type that explicitly lists all of the possible icon names.
Then we'll have to keep the icon names in sync with their image files on disk.
That type might look like:
type IconName =
"menu" |
"arrowDown" |
"arrowLeft" |
"arrowRight" |
...
In English: "a value of type IconName is statically guaranteed to be one of the strings specified here, but not any other string."
(This type is a combination of two topics covered by Execute Program lessons: literal types and type unions.)
Our IconName isn't defined as a simple union of literal string types.
Keeping a list of icon names in sync with a list of files is boring work that we can make the computer do!
Instead, our icon.tsx file looks like this:
export const icons = {
arrowDown: {
label: "Down Arrow",
data() {
return <path ... />
}
},
arrowLeft: {
label: "Left Arrow",
data() {
return <path ... />
}
},
...
}
The actual SVG <path /> tags are right inside the source code, in an object keyed by the icon's name.
(It's also possible to do this without inlining the SVG into a source file.
For example, we could use some Webpack tricks to keep the images in their own files, but still have a guarantee that every icon in the list also exists on disk.
So far, this simpler solution has worked well for us.)
By defining the icons in this way, we can extract a union type of their names automatically with one line of code:
export type IconName = keyof typeof icons
(In English, you can think of that type as saying "whenever something has the type "IconName", it must be a string that matches one of the keys of the icons object.)
That's it; there's no other type-level work required.
The rest of the code is just a straightforward Icon React component that looks up an icon in a list and returns its SVG path.
There are no explicit TypeScript types in that function; it looks like pure JavaScript, though it's still type-checked.
Here's a minimal version with all unrelated details stripped away:
export function Icon(props: {
name: IconName
}) {
return <svg>
{icons[props.name].data()}
</svg>
}
Now we can drop new icons into the "icons" list by putting the SVG tags in that source file. When we do that, the icon becomes available for use in the Button component, as well as any other part of the system that accepts an icon name. If we delete an icon from the list, every part of the system that references it instantly fails to compile, ensuring that we have no stale icon references that can cause errors at runtime.
These examples are simple by static type standards, but I think that they illustrate how much low-hanging fruit there is in a web application. Most of an application's code doesn't involve advanced type system features; it's simple stuff like "make sure that we're passing the right props" and "make sure that our icons actually exist."
We do this kind of thing all over the system. Some more examples:
Note component used throughout the system.
It has a tone prop to determine the style of the note: "info", "warning", "error", etc.
If we retire one of those tone options, we'll remove it from the union type, and all Note uses that referenced that tone will error until we update them.Problems like these come up often in programming, especially in dynamic languages, but can be statically prevented without writing any automated tests and without doing any manual testing. Some of them take work; our API router verification was tricky to write. But a lot of them are easy. The one-line "IconName" type above really is the entire solution to the problem; it will work if you copy it into a TypeScript file.
Porting our frontend code to TypeScript was just the beginning. We've since ported the backend from Ruby to TypeScript, then grown and maintained it for nine months after the port.
Gary Bernhardt
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) [9] => Array ( [uri] => https://www.executeprogram.com/blog/why-you-should-learn-sql [title] => Why You Should Learn SQL [timestamp] => 1572558204 [author] => [content] =>The SQL language is old, strange, and important. We'll take those in reverse order:
SQL is important. Unlike many other types of databases, SQL databases can give strong guarantees of data correctness.
If we have a regular, user-facing bug in our application, some users will see an error. When we fix the bug, the users stop seeing the error and everything is fine.
But if we have a bug that inserts incorrect data into the database, then fixing the bug doesn't fix the problem. Even after the fix, the database is full of incorrect data, so we have to do something to clean it up. Here's a concrete example:
We never expected orders to have a null phone_number, but a temporary application bug inserted some nulls there. Now our order history page is rendering those orders incorrectly. The business reports used by the management team throw errors because they don't expect nulls, so the business's cashflow is now invisible. Our analytics continue to run, but they produce incorrect numbers because every order with a null phone_number is defaulting to country code 1 (the USA and Canada).
There's no way to retroactively get correct phone numbers for all of those orders!
We have two possible ways to mitigate this problem once it's happened. First, we can adapt the rest of the system to correctly handle the nulls. That's a lot of work to handle a case that was never supposed to happen; we're only considering this because of a bug we already fixed.
The second option is to make up fake phone numbers for all of the affected orders. Said another way: we can intentionally put incorrect data into the database, complicating attempts to contact those customers.
Neither of those options is desirable. However, there is a third option: tell the database that the phone_number is never allowed to be null.
In a SQL database like PostgreSQL, that non-nullability is guaranteed. If we try to insert a null, the database rejects it. If PostgreSQL allows us to insert a null in that column, it's a bug in PostgreSQL itself. (PostgreSQL is stable and well-tested; you should never expect to find a bug like this.)
OK: SQL is important because our data is important. It's also very strange, but in ways that make it powerful.
Most popular languages are imperative: code starts at the top of the function and executes line by line until the bottom. This is our default for a good reason: it's easy to think about how the code executes, at least within a single function.
SQL isn't like that at all. In SQL, we describe the results that we want declaratively. For example, in an imperative language we might say: "Loop over each cat. If a cat's age is over 5, add it to the oldCats array. Then return the oldCats array." In SQL, we say something more akin to "Give me all of the cats whose age is over 5."
What happens when we execute these two versions of the code? The imperative version says what the computer should do, step by step. It's difficult for a compiler to optimize it because it's so rigid.
The SQL version doesn't say what order things should happen in; it just describes the result. That lets the database optimize it aggressively. The database will slice up the query into its smallest parts, reorder it as needed, throw some parts away if they're not needed, replace them with more efficient versions, etc. This all happens without any input from us as the programmers, and it's guaranteed not to change the results.
In everyday situations, that optimization turns minutes-long queries into single-digit milliseconds. In extreme cases, it turns years-long queries into seconds.
Compilers for imperative languages have optimizers too. But SQL databases' optimizers are much more powerful because SQL lets us say what data we want, rather than how to get it.
OK: SQL is important and strange. It's also old: it first appeared in 1974. That definitely shows, so let's not kid ourselves. In SQL, we say things like "ALTER TABLE users ALTER COLUMN email DROP NOT NULL".
In English, that means "allow users' email addresses to be null from now on". The SHOUTING PARTS are SQL syntax, whereas "users" and "email" are parts of our data. This shouting convention comes from THE OLD TIMES.
You can see that SQL is unusually wordy. JavaScript has 64 keywords: "if", "function", "for", etc. C has 34. SQLite has 140 keywords. But SQLite is "lite", like its name says. PostgreSQL 11 has 760 keywords. That's the exact number of words that you've read in this article so far, not including this sentence!
OK, so SQL is important, which makes it worth learning. But it's also strange and old, which can make it offputting.
That's a great reason for us to teach Execute Program to teach you SQL. We build complex knowledge models of tools like SQL, regular expressions, and TypeScript, which allows Execute Program to teach the topics in thousands of different orders depending on your strengths. Unlike most learning tools, you'll rarely go a full minute without completing an interactive code example. Those examples are then reviewed automatically, on exponentially increasing intervals, so you don't forget them.
You can try our first SQL lesson without creating an account. It starts lightly, at the very beginning, and ramps up from there. Our favorite lesson comes late in the course, when you write a real SQL injection exploit to attack your own database. We hope that you enjoy it!
Gary Bernhardt
[enclosures] => Array ( ) [categories] => Array ( ) [uid] => ) ) )