From 62fe16623ba79e3961b64f34243bb63adb1fc648 Mon Sep 17 00:00:00 2001 From: Ken Date: Tue, 26 May 2026 23:11:49 +0000 Subject: [PATCH] feat: migrate existing posts to Astro content collection Copy two existing blog posts from content/posts/ (old Gatsby location) to src/content/posts/ with updated frontmatter: - localize-react-without-bloating-the-bundle.md - speeding-up-webpack-typescript-incremental-builds-by-7x.md (was .mdx) Frontmatter changes: removed path/heroImage fields, added type/tags/summary, simplified date to YYYY-MM-DD format. Body content preserved verbatim. --- ...alize-react-without-bloating-the-bundle.md | 110 +++++++++++++ ...ack-typescript-incremental-builds-by-7x.md | 147 +++++++++++++++++ tests/migrate-posts.test.mjs | 155 ++++++++++++++++++ 3 files changed, 412 insertions(+) create mode 100644 src/content/posts/localize-react-without-bloating-the-bundle.md create mode 100644 src/content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.md create mode 100644 tests/migrate-posts.test.mjs diff --git a/src/content/posts/localize-react-without-bloating-the-bundle.md b/src/content/posts/localize-react-without-bloating-the-bundle.md new file mode 100644 index 0000000..c82107d --- /dev/null +++ b/src/content/posts/localize-react-without-bloating-the-bundle.md @@ -0,0 +1,110 @@ +--- +title: 'Localize React without Bloating the Bundle' +date: 2019-08-16 +type: post +tags: [react, localization, webpack, performance] +summary: 'A performant approach to React localization using dynamic imports and react-intl-universal, keeping your main bundle free of locale strings.' +--- + +There are so many possibilities when you want to localize your application with your React application. I believe that localization is difficult because it requires excellence in several pieces of a stack at the same time. You have to have a working pipeline that can take string resources that would get translated. Then, you have to have a way to load these strings onto your page or application. Finally, you have to have a way to take these strings and inject them into the components inside your application. As you can see, you can choose just about any tech to help you accomplish these goals. I am documenting a particular set of stack that I believe helps you achieve this in the most performant way with tools that you're already probably using. + +## Translation Pipeline + +At work, we have a translation-as-a-service API that we rely on to refresh localized strings for us every night. There's a growing team that has built a simple Azure DevOps task we add as a step in one of our pipelines which runs nightly. + +Not everyone is as fortunate that has something they can use from their own company. Given that, I'll suggest a pattern here as the first step. Do a search for "localization as a service" and look for a vendor that can help add a step in your CI pipeline of choice. Set up a nightly job to refresh your application's localized strings as JSON like this: + +```json +{ + "HELLO_NAME": "Hello {name}!", + "CLICK_ME": "Click me" +} +``` + +## Rendering Localized Strings + +React is a large ecosystem. So then the paradox of choice is real when finding supplemental libraries for React. Conventional wisdom is to find the most popular packages from npmjs.org. So given this, I first looked at react-intl to help me inject localized strings into my application. The issue here is that react-intl uses higher order components all over the place. One of the explicit goals (as I heard it from ReactConf 2018) of React hooks is to do away with depth of the component tree caused by the higher order components. Higher order component, or HOC, is a neat idea until the consumer needed to access the ref to the original wrapped component. When all your components that use localized strings are wrapped in HOCs, your application start to look like a sideways mountain. (aside: go look at your component tree in React DevTool to see if you're suffering from HOC-itis) + +Enter react-intl-universal. The Alibaba Group created this library to get around the HOC issues of react-intl. On top of this, there are times where strings are needed from outside of the component's render() method. It takes 2 steps to place strings inside your components. + +First you have to initialize the locale data. Note that the data can be preloaded from a server or can be retrieved at runtime. The choice is yours. For the most optimal case, we definitely would have the server preload strings right in the app as it is being loaded. + +Let's pretend that ./locales/en-US.json has the same content as the example above. + +```js +// locale data +const locales = { + "en-US": require("./locales/en-US.json"), + "zh-CN": require("./locales/zh-CN.json"), +} +``` + +Then, we initialize the react-intl-universal library inside a componentDidMount() call. And we'll use the localized string inside the render() method with the .get() function: + +```js +import intl from "react-intl-universal" + +class App extends Component { + state = { isLoading: false } + componentDidMount() { + this.loadLocales() + } + + async loadLocales() { + await intl.init({ + currentLocale: "en-US", + locales, + }) + this.setState({ isLoading: true }) + } + + render() { + return ( + !this.state.isLoading && ( +
{intl.get("HELLO_NAME", { name: "world" })}
+ ) + ) + } +} +``` + +Note that the init() call returns a Promise. This means that we can use the async / await syntax to write our string load code. Once this is added, we look at the way we retrieve the strings by key. For that, we use the get(). Get takes in two parameters: the key and some object. Sometimes the strings have slots that can be replaced by the object values. + +## Loading Localized Strings + +This is where it gets interesting. So far, we've assumed that we had the locale data all upfront. This means that all the localized strings would had been loaded inside a bundle or onto the page somehow. Loading all the language strings in one go can only be feasible if the app barely contain any text. If we're using Webpack, we should take advantage of a feature that I recently came to know. We all have seen the dynamic import() syntax: + +```js +const SomeModule = import("some-module") +``` + +But, have you seen what Webpack can do with something like this? + +```js +const getLocale = locale => import(`./locales/${locale}.json`) +``` + +Based on the .json files it finds inside ./locales, Webpack is smart enough to generate chunks for dynamic loading! That means your main bundle will not incur the weight of the entire library of localized strings. Putting all these concepts together, I've created a repo to demonstrate concepts from this post: + +https://github.com/kenotron/react-intl-example + +I'll go over some of the points from that repo. First, I created a HOC that you place at the ROOT of the application. Don't worry! It is only one HOC for the entire app. It is called LocaleComponent - I'm keeping this strange little name until React.createResource() becomes a thing maybe in the future. + +```js +const getLocale = locale => import(`./locale/${locale}.json`) + +class LocaleComponent extends React.Component { + state = { isLoading: true } + + async loadLocales() { + const locales = await getLocale("en") + const currentLocale = "en" + await intl.init({ currentLocale, locales }) + this.setState({ isLoading: false }) + } + + render() { + return !this.state.isLoading ? <>this.props.children : null + } +} +``` diff --git a/src/content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.md b/src/content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.md new file mode 100644 index 0000000..54b3d8b --- /dev/null +++ b/src/content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.md @@ -0,0 +1,147 @@ +--- +title: 'Speeding Up Webpack Typescript Incremental Builds by 7x' +date: 2019-08-16 +type: post +tags: [webpack, typescript, performance, microsoft] +summary: 'How the Outlook team at Microsoft cut incremental build times from 35s to 5s by defeating four performance enemies in the Webpack pipeline.' +--- + +What if I told you your Webpack is doing too much work all this time? Webpack 4 brought a lot of goodies for developers but to use it at scale, the Outlook team at Microsoft had to take a hard look at the incremental build numbers to find out. Here’s how we made our incremental builds go from 35s to a consistent 5s. + +I guess it goes without saying that you MUST measure in order for you to know you have made progress! + +Let’s name some enemies of incremental build speed: + +1. stats.toJson() +2. Competing resolution logic between Webpack and its loaders (ts-loader) +3. Garbage Collection +4. Subtle v8 ES6 perfomance issues +5. The Base Line + +To begin, there are already a few things beyond setting the mode in webpack.config.js we already apply so we’re not doing too much optimization during incremental builds: + +```json +{ + "optimization": { + "removeAvailableModules": false, + "removeEmptyChunks": false, + "splitChunks": false + } +} +``` + +Alright, so let’s establish the baseline by looking at a typical inner loop flame graph: + +As you can see, we’re clocking in at around 40s here per incremental build. This is not exactly true because we lose about 5s of it due to profiling. In measuring with our internal telemetry, we noticed that our devs are hitting around 30–35s on avg (and sometimes over a minute at the 75th percentile) incremental builds. + +So, as soon as you look at those colors, you would recognize three separate phases of the incremental build process. With this in mind, let’s tackle the first enemy. + +## Enemy #1: stats.toJson is VERY heavy in WP4 + +If you were looking at the the CPU profile flame graph, you would notice that the last phase of the process is dominated by a bunch of stats.toJson calls. Where does it come from? It’s right inside webpack-dev-server’s Server.js: + +```js +const clientStats = { errorDetails: false }; +... +comp.hooks.done.tap('webpack-dev-server', (stats) => { +this.\_sendStats(this.sockets, stats.toJson(clientStats)); +this.\_stats = stats; +}); +``` + +The issue here is that Webpack 4 gave `toJson` a lot more information, but it also regressed the performance tremendously as a result. The fix is in a pull request: + +https://github.com/webpack/webpack-dev-server/pull/1362 + +This is the big one — it brought our incremental speeds from 30s to around 15s. + +> Update: the webpack-dev-server maintainers had accepted my patch! So, go ahead and use webpack-dev-server@3.1.2. Personally, I have observed a slight 0.5s regression between 2.x release and the 3.x release, so we’re keeping the 2.x for now until we can move to using webpack-serve. + +Since we’re waiting for the authors to merge this for v2, I’ve published a temporary fork for it for the v2 branch: + +https://www.npmjs.com/package/webpack-dev-server-speedy + +For the fix on v2, you’ll have to use the node API to take advantage of it like this package in your build process: + +```js +const Webpack = require("webpack") +const WebpackDevServer = require("webpack-dev-server-speedy") +const webpackConfig = require("./webpack.config") +const compiler = Webpack(webpackConfig) +const devServerOptions = Object.assign({}, webpackConfig.devServer, { + stats: { + colors: true, + }, +}) + +const server = new WebpackDevServer(compiler, devServerOptions) + +server.listen(8080, "127.0.0.1", () => { + console.log("Starting server on http://localhost:8080") +}) +``` + +## Enemy #2: Competing resolution logic between Webpack and ts-loader + +If I were to ask you to build an incremental compiler based on Typescript, you would likely first reach into the Typescript API for something that it is using for its watch mode. For the longest time, Typescript safe guarded this API from external modules. During this time, ts-loader was born. The author of the loader tracked the progress of another Typescript-centric loader called awesome-typescript-loader and brought back the idea of doing type checking on a separate thread. This transpileOnly flag worked remarkably well (with a rather glaring caveat that const enums are not supported out of the box — here’s a workaround from the ts-jest repo) until the codebase reaches a certain size. + +In OWA, we have nearly 9000 modules that we shove across this loader. We have found that the first phase of that incremental build is linearly growing as our repo grows. + +Things looked pretty grim until the Typescript team decided to take on this mammoth work of expose the watch API to external modules. Specifically, after this was merged, ts-loader is super charged with the ability to limit the amount of modules to transpile at a time per iteration! + +We just add this to our webpack config module.rules: + +```json +{ + "test": /\.tsx?\$/, + "use": [ + { + "loader": "ts-loader", + "options": { + "transpileOnly": true, + "experimentalWatchApi": true + } + } + ] +} +``` + +Don’t forget to the typechecker when appropriate: https://www.npmjs.com/package/fork-ts-checker-webpack-plugin (we have a mode to turn type checker OFF for even faster rebuilds) + +The incremental builds now only rebuilds around 30–40 modules rather than 50% of our modules! I also have a way to CAP the growth of the incremental builds in the first phase. + +This optimization cuts our 15s to around 8s. + +## Enemy #3: Garbage Collection + +So Garbage Collection is a great invention. But not in a tight loop. Perhaps there’s a perf bug inside node or v8, but I’ve discovered that a global string.replace(/…/g, ‘…..’) can cause a lot of GC when placed inside a loop. Webpack 4 introduced the path info in the generated dev mode replacing module ids with more useful path info. This is done with, you guessed it, global string replace with regex. It then created a LOT of unnecessary GCs along the way. (as an aside, perhaps I should file a bug against either Webpack, node, or v8…) + +Okay, let’s turn that sucker off in webpack.config.js in the output.pathinfo: + +```json +{ + "output": { + "pathinfo": false + } +} +``` + +Just ask yourself if you REALLY need that pathinfo or that build speed. For us, we chose speed. This made our 8s builds to around 6s + +## Enemy #4: Subtle v8 ES6 perfomance issues + +Most everyone would be pleased with that 6s figure, but why should we humans not demand MOAR? Yes, MOAR speed!!! + +In chatting with a colleague of mine, John-David Dalton, about his project, esm, he told me about node.js performance issues with ES6 data structures like Map and Set. Having dug into Webpack source code previously and by looking at the remaining profile slowdowns (looking at the “heavy” or “bottom-up”), I noticed that Webpack’s internal algorithm is dominated by calling their SortableSet methods. Since SortableSet extends Set, it would follow that Webpack is actually greatly affected by the speed of the Map/Set implementation of V8. Here’s the bug: + +https://github.com/nodejs/node/issues/19769 + +So, I advise everyone doing heavy Webpack development to switch to the LTS (v10+ or stick with v8.9.4) + +Using that version, the incremental build is down to 4.5s + +## Why? Inventing on Principle! + +Finally, I want to leave you with the best motivation on why we should reduce this incremental build speeds down to almost nothing: + +Hey! follow me on twitter @kenneth_chau to get more articles like these :) diff --git a/tests/migrate-posts.test.mjs b/tests/migrate-posts.test.mjs new file mode 100644 index 0000000..9878320 --- /dev/null +++ b/tests/migrate-posts.test.mjs @@ -0,0 +1,155 @@ +import { describe, it } from 'node:test'; +import { strict as assert } from 'node:assert'; +import { readFileSync, existsSync } from 'node:fs'; +import { resolve } from 'node:path'; + +const postsDir = resolve('src/content/posts'); + +const post1Path = resolve(postsDir, 'localize-react-without-bloating-the-bundle.md'); +const post2Path = resolve(postsDir, 'speeding-up-webpack-typescript-incremental-builds-by-7x.md'); + +const originalPost1Path = resolve('content/posts/localize-react-without-bloating-the-bundle.md'); +const originalPost2Path = resolve('content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.mdx'); + +function extractFrontmatter(content) { + const match = content.match(/^---\n([\s\S]*?)\n---/); + return match ? match[1] : null; +} + +function extractBody(content) { + const match = content.match(/^---\n[\s\S]*?\n---\n([\s\S]*)$/); + return match ? match[1] : null; +} + +describe('Migrated post: localize-react-without-bloating-the-bundle', () => { + it('should exist at src/content/posts/localize-react-without-bloating-the-bundle.md', () => { + assert.ok(existsSync(post1Path), 'Post file should exist'); + }); + + describe('Frontmatter', () => { + it('should have title field', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /title:\s*['"]?Localize React without Bloating the Bundle['"]?/, 'should have correct title'); + }); + + it('should have date as 2019-08-16 (YYYY-MM-DD)', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /date:\s*2019-08-16\s*$/m, 'should have simplified date'); + }); + + it('should have type: post', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /type:\s*post/, 'should have type: post'); + }); + + it('should have correct tags', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /tags:/, 'should have tags field'); + assert.match(fm, /react/, 'should include react tag'); + assert.match(fm, /localization/, 'should include localization tag'); + assert.match(fm, /webpack/, 'should include webpack tag'); + assert.match(fm, /performance/, 'should include performance tag'); + }); + + it('should have summary field', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /summary:/, 'should have summary field'); + }); + + it('should NOT have path field', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.doesNotMatch(fm, /^path:/m, 'should not have path field'); + }); + + it('should NOT have heroImage field', () => { + const content = readFileSync(post1Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.doesNotMatch(fm, /heroImage:/, 'should not have heroImage field'); + }); + }); + + describe('Body content', () => { + it('should have body matching original post verbatim', () => { + const newContent = readFileSync(post1Path, 'utf-8'); + const originalContent = readFileSync(originalPost1Path, 'utf-8'); + const newBody = extractBody(newContent); + const originalBody = extractBody(originalContent); + assert.equal(newBody, originalBody, 'Body content should match original exactly'); + }); + }); +}); + +describe('Migrated post: speeding-up-webpack-typescript-incremental-builds-by-7x', () => { + it('should exist at src/content/posts/speeding-up-webpack-typescript-incremental-builds-by-7x.md', () => { + assert.ok(existsSync(post2Path), 'Post file should exist'); + }); + + it('should be a .md file (not .mdx)', () => { + assert.ok(post2Path.endsWith('.md'), 'should be .md extension'); + assert.ok(!post2Path.endsWith('.mdx'), 'should not be .mdx extension'); + }); + + describe('Frontmatter', () => { + it('should have title field', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /title:\s*['"]?Speeding Up Webpack Typescript Incremental Builds by 7x['"]?/, 'should have correct title'); + }); + + it('should have date as 2019-08-16 (YYYY-MM-DD)', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /date:\s*2019-08-16\s*$/m, 'should have simplified date'); + }); + + it('should have type: post', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /type:\s*post/, 'should have type: post'); + }); + + it('should have correct tags', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /tags:/, 'should have tags field'); + assert.match(fm, /webpack/, 'should include webpack tag'); + assert.match(fm, /typescript/, 'should include typescript tag'); + assert.match(fm, /performance/, 'should include performance tag'); + assert.match(fm, /microsoft/, 'should include microsoft tag'); + }); + + it('should have summary field', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.match(fm, /summary:/, 'should have summary field'); + }); + + it('should NOT have path field', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.doesNotMatch(fm, /^path:/m, 'should not have path field'); + }); + + it('should NOT have heroImage field', () => { + const content = readFileSync(post2Path, 'utf-8'); + const fm = extractFrontmatter(content); + assert.doesNotMatch(fm, /heroImage:/, 'should not have heroImage field'); + }); + }); + + describe('Body content', () => { + it('should have body matching original post verbatim', () => { + const newContent = readFileSync(post2Path, 'utf-8'); + const originalContent = readFileSync(originalPost2Path, 'utf-8'); + const newBody = extractBody(newContent); + const originalBody = extractBody(originalContent); + assert.equal(newBody, originalBody, 'Body content should match original exactly'); + }); + }); +}); \ No newline at end of file