Journey to webpack, part 2 - One small step

If you missed the first part of this journey, pop over to Part 1.

OK, let’s get rolling here and jump right in…. let’s convert everything to use webpack and reap the benefits!

YES!

Me: “Let’s start easy…. OK, how do I concat then minify then…” Me: 10 minutes later: “Oh shit, there is no thenmind blown

OK, so webpack is a completely different mindset to the task-runner-approach I am familiar with like Make/Rake/Grunt/Gulp, heck even Ant. Different doesn’t mean bad (that’s what I tell my kids!), but this isn’t going to be a quick transition.

So where do we start, for real?

We’ve had Grunt tasks in place for years (literally) so we already create minified bundles. These are the main ones:

We decided that the first step should be to attack our 3rd party dependencies - namely anything in node_modules/ that we didn’t create. We still call it “vendor” because we used to use Bower for everything. Then we used a mix of Bower and npm, and finally now we are 100% npm (well some of us moved to yarn, but npm v5 looks sweet… I digress). But the name “vendor” has stuck, and I think that’s OK.

Aside: Bower is a good reminder that all software has a shelf life. Nothing lasts forever in this crazy fast changing world of software. I don’t know what will replace webpack, but we’re being diligent about our inputs and outputs, and we should have an easier time transitioning to the next thing because we are more educated on just what exactly is in our project. Also, in a weird way, the webpack config seems kinda self documenting, which is awesome.

We settled on vendor.min.js because:

  1. It’s all 3rd party stuff that has hopefully already been webpack’d by someone
  2. It’s all JavaScript (no TypeScript to worry about… yet)
  3. We could literally replace the “uglify:vendor” task with a new task named “shell:webpack”

By approaching it this way, we could immediately merge into staging (our production line) without needing to wait until we have the entire project converted. We were fearful this could take weeks/months to fully complete and wanted to show progress… and also feel good about ourselves. baby steps

webpackify the vendors

If you’re like us, you have more 3rd party dependencies than you’d like to admit. It’s cool. We all do it. We’re trying to be more stringent now, but back when we started, we’d pull in anything and everything. Hence we have 37 distinct JavaScript files we rely on. Yup. 37. Stop me if this sounds familiar: modernizr, detectizr, jQuery, lodash, moment, d3, bootstrap, fastclick, angular, angular-cookie, angular-cookies (wha?), angular-animate, angular-ui-router, angular-moment, angulartics, isotope, phoneformat, bootstrap-colorpickers, highcharts, ng-sortable, ocLazyLoad, log4javascript… you get the idea. Honestly at this point it doesn’t matter if it’s 2 or 22, we have a bunch of files from other places that we need to concat & minify into a single file named vendor.min.js. This isn’t rocket science. Let’s see how we’d do it in webpack.

here we go

First, we specify where we want this file to go. That looks easy enough: there is a config option named “output” that can take a “filename”. We know where we want it to end up and what to call it so:

module.exports = {
output: {
filename: ‘build/static/app/vendor/js/vendor.min.js’
},
}

Now, how do we tell webpack to “go grab all these things in node_modules/ and do your thing”? webpack has this concept of an “entry point”. Sweet! I can just point it at the entry point of my application and … DOH! Wait a sec. I can’t do that. If I do that, it’s going to try to do everything, I want to do this systematically and only work on node_modules/ … Oh and did I mention that I don’t have an actual entry point? This is a Angular 1.5 application built with a mix of JavaScript and TypeScript. Shame on me. I thought our entry point might be index.html! Clearly pointing webpack at node_modules/ wasn’t going to work (yeah that was my first hope). Even if somehow it did work, we’d have to filter out anything that we’ve written ourselves (what I call 2nd party libraries), plus all the junk that isn’t used at runtime… yeah it’d be a mess.

create your own entry point

It’s fun working with smart people. This time we leaned on Tor, who has a little webpack experience. He’s getting credit for this whether he figured it out himself, read it somewhere or just took a wild guess (I’m betting on the first one). Say hello to the “entry point file”. Basically what we will do is create a new file that imports all the files that we want in the bundle. In other words, we’re faking it ‘til we’re making it…. and Tor approves. [caption id=”” align=”aligncenter” width=”240”] Tor to the rescue![/caption] So we build up a file called vendor.ts that looks like this:

import ‘../node_modules/modernizr/modernizr’;
import ‘../node_modules/detectizr/dist/detectizr’;
import ‘../node_modules/jquery/dist/jquery’;
import ‘../node_modules/jquery.easing/jquery.easing’;
import ‘../node_modules/lodash/index’;
import ‘../node_modules/moment/min/moment-with-locales.min’;
import ‘../node_modules/moment-timezone/builds/moment-timezone-with-data’;
import ‘../node_modules/d3/d3’;
import ‘../node_modules/bootstrap/dist/js/bootstrap’;
import ‘../node_modules/fastclick/lib/fastclick’;

Then we update our webpack.config.js to use it as an entry point:

module.exports = {
output: {
filename: ‘build/static/app/vendor/js/vendor/min.js’
},
entry: {
vendor: ‘./webpack-config/vendor.ts’,
},
}

Let’s do this! At this point, we’ve created our webpack Grunt task, so we can just run “grunt shell:webpack”…

Running “shell:webpack” (shell) task
Hash: 4580ce86f289c11a996b
Version: webpack 2.6.1
Time: 4245ms
Asset Size Chunks Chunk Names
build/static/redbox/vendor/js/vendor.min.js 5.15 MB 0 [emitted] [big] vendor
build/static/redbox/vendor/js/vendor.min.js.map 6.25 MB 0 [emitted] vendor
[1] .//jquery/dist/jquery.js 248 kB {0} [built]
[112] ./
/intl-tel-input/lib/libphonenumber/build/utils.js 217 kB {0} [built]
[113] .//isotope/dist/isotope.pkgd.js 103 kB {0} [built]
[114] ./
/jquery-form/jquery.form.js 42.5 kB {0} [built]
[115] .//jquery.easing/jquery.easing.js 4.87 kB {0} [built]
[116] ./
/lodash/index.js 411 kB {0} [built]
[117] .//log4javascript/log4javascript.js 128 kB {0} [built]
[118] ./
/modernizr/modernizr.js 51.4 kB {0} [built]
[119] .//moment-timezone/builds/moment-timezone-with-data.js 193 kB {0} [built]
[120] ./
/moment/min/moment-with-locales.min.js 150 kB {0} [built]
[121] .//ng-sortable/dist/ng-sortable.js 38.9 kB {0} [built]
[122] ./
/oclazyload/dist/ocLazyLoad.js 57.7 kB {0} [built]
[123] .//phoneformat/phoneformat.min.js 433 kB {0} [built]
[124] ./
/tslib/tslib.js 6.47 kB {0} [built]
[127] ./webpack-config/vendor.ts 2.08 kB {0} [built]
+ 113 hidden modules

SUCCESS! We’ve got a vendor.min.js that weighs in at a hefty 5MB (yup, this is before we uglify… patience my friend), it’s in the right place, and now we should be able to start up our application!

make it part of the build

Next we replace the Grunt task “uglify:vendor” with “shell:webpack” and cross our fingers while our dev server starts up. Fire up the browser, and … [caption id=”attachment_731” align=”aligncenter” width=”1062”] vendor.min.js:108094 Uncaught TypeError: Cannot read property ‘documentElement’ of undefined
index.html:35 Uncaught ReferenceError: angular is not defined[/caption] That’s weird. After a quick double check, angular.js is in fact in our entry point file and, because we didn’t yet uglify, I can actually see it in vendor.min.js. Not sure what to do next, we decided - “OK if they don’t ALL work, let’s try paring them down”. This required putting back the “uglify:vendor” task and commenting out only the inclusion of modernizr & detectizr. Back in vendor.ts, we remove everything but modernizr & detectizr. We also had to modify our setup slightly. We’ll have Grunt continue to build vendor.min.js and we will have webpack build a new file called vendor-webpack.min.js. Finally, we’ll modify index.html to load vendor-webpack.min.js before vendor.min.js. PHEW!

Hash: a76a5e5b36fcdc8a8181
Version: webpack 2.6.1
Time: 187ms
Asset Size Chunks Chunk Names
build/static/redbox/vendor/js/vendor-webpack.min.js 73.7 kB 0 [emitted] vendor
build/static/redbox/vendor/js/vendor-webpack.min.js.map 87.4 kB 0 [emitted] vendor
[0] .//detectizr/dist/detectizr.js 16.8 kB {0} [built]
[1] ./
/modernizr/modernizr.js 51.4 kB {0} [built]
[2] ./webpack-config/vendor.ts 2.08 kB {0} [built]

Fire up the browser, and … [caption id=”attachment_732” align=”alignnone” width=”914”] Uncaught TypeError: Cannot read property ‘detect’ of undefined[/caption] What now? I think this is better, but WTF?

to the google

Having really no clue what’s going on, we took to Google and quickly learned that we need to let webpack know about any libraries that expose global variables. This is done with the expose-loader. OK, this isn’t so bad:

yarn add expose-loader

Since both modernizr and detectizr expose globals, we’ll add both to webpack.config.js:

module: {
rules: [
{
test: require.resolve(‘modernizr’),
use: [{
loader: ‘expose-loader’,
options: ‘Modernizr’
}]
},
{
test: require.resolve(‘detectizr’),
use: [{
loader: ‘expose-loader’,
options: ‘Detectizr’
}]
}
]
}

Fire up the browser, and … [caption id=”attachment_733” align=”aligncenter” width=”706”] Uncaught TypeError: Cannot read property ‘documentElement’ of undefined[/caption] Hmmm. That looks familiar. It’s the same error we had before, but it seems that window.Modernizr is defined at least. And we don’t see the error about Detectizr.detect. In the devtools console we can see that both Modernizr and Detectizr are globally available. We’ll call this progress!?

fixing the ‘izrs

Again, not really having any clue about where to go next, we check Google and find some links specific to modernizr. As much as I hate the copy-paste-pray method, sometimes it’s all you got.

yarn add imports-loader exports-loader

So we blindly add to our configuration:

module: {
rules: [
{
test: require.resolve(‘modernizr’),
use: [{
loader: ‘expose-loader’,
options: ‘Modernizr’
}, {
loader: ‘imports-loader?this=>window!exports-loader?window.Modernizr’
}]
},
{
test: require.resolve(‘detectizr’),
use: [{
loader: ‘expose-loader’,
options: ‘Detectizr’
}, {
loader: ‘imports-loader?this=>window!exports-loader?window.Detectizr’
}]
}
]
}

Fire up the browser, and … No errors! It works! Commit! Push! Call it a day! Well not so quick. What the heck is this thing doing?

the import/export business

Here we learn that many NodeJS modules are written to expect “this” to be the global space, and browser’s call the global space “window”. So what the imports-loader does is transform “this” to “window” with the “this=>window” syntax. Read more about shimming.

Let’s break this down a bit:

loader: ‘imports-loader?this=>window!exports-loader?window.Modernizr’
|————|||———-|||————|||————–|
| | | | | | |
| here | loader | here |
| come some | separator | come some |
| parameters | | parameters |
| | | |
loader the loader the
name params name params
to the loader to the loader

Aside: This is probably the last time I try to do an ascii diagram by hand. If you are on a phone, I am sorry, it’s kind of a mess. Admittedly it was fun to make. By hand. With vim. Yup. This little shorthand is intimidating at first, but becomes more clear when you use it the second time. Now, if you’re paying attention, it seems we are both “exporting” and “exposing” window.Modernizr. What’s up with that? Do we need both? Well, it certainly works with both, let’s try without expose-loader - Running webpack results in this error:

ERROR in ./~/modernizr/modernizr.js
Module parse failed: /Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/imports-loader/index.js?this=>window!exports-loader?window.Modernizr!/Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/modernizr/modernizr.js Unexpected token (1411:13)
You may need an appropriate loader to handle this file type.
| })(this, this.document);
|
| }.call(window!exports-loader?window.Modernizr));
@ ./webpack-config/vendor.ts 1:0-45

ERROR in ./~/detectizr/dist/detectizr.js
Module parse failed: /Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/imports-loader/index.js?this=>window!exports-loader?window.Detectizr!/Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/detectizr/dist/detectizr.js Unexpected token (507:13)
You may need an appropriate loader to handle this file type.
| }(this, this.navigator, this.document));
|
| }.call(window!exports-loader?window.Detectizr));
@ ./webpack-config/vendor.ts 2:0-50

OK, so it’s needed to even run webpack, but it’s not entirely clear why. Well, let’s try to consolidate these loaders. The shorthand syntax is pretty sweet, so let’s append the expose-loader and just make this a one-liner:

loader: ‘imports-loader?this=>window!exports-loader?window.Modernizr!expose-loader?Modernizr’

Ugh. That didn’t work. We get the same build error as above. According to the documentation, webpack v2 doesn’t support chaining this way, so maybe we shouldn’t head down this road. Let’s back up a bit and clearly define the 3 things we know need to happen:

{ 
  test: require.resolve('modernizr'),
  use: \[{
    loader: 'expose-loader',
    options: 'Modernizr'
  }, {
    loader: 'imports-loader',
    options: 'this=>window'
  }, {
    loader: 'exports-loader',
    options: 'window.Modernizr'
  }\]
},

Ordering here is important. webpack processes this array bottom->top. We need the exports-loader to run before imports-loader. I still haven’t figured out why we need both the expose-loader and the exports-loader, but we’re back to working again. It seems it would only be necessary if you were using an import or require. Maybe it’s because we’re importing it in vendor.ts? I figured if we exposed it, that would be enough, because it seems all that is needed is for the variable to be available globally. (TODO: Update when I figure out why) Now let’s see if we can pass the “options” as parameters to the loader like in webpack v1

{
  test: require.resolve('modernizr'),
  use: \[{
    loader: 'expose-loader?Modernizr'
  }, {
    loader: 'imports-loader?this=>window'
  }, {
    loader: 'exports-loader?window.Modernizr'
  }\]
},

As expected, this also works. And by the way, I am loving the short-hand!

clean up in aisle 2

If we go back & look at the documentation, webpack v2 will also accept an array of loaders in the “use” option. So we can make this even more pleasing to the eye with:

{
test: require.resolve(‘modernizr’),
use: [
‘expose-loader?Modernizr’,
‘imports-loader?this=>window’,
‘exports-loader?window.Modernizr’
]
},
{
test: require.resolve(‘detectizr’),
use: [
‘expose-loader?Detectizr’,
‘imports-loader?this=>window’,
‘exports-loader?window.Detectizr’
]
}

Now we’re talking!

git commit -am “webpack: vendor-webpack.min.js contains modernizr and detectizr” && git push origin HEAD

one giant leap

We’ll end here for today. We need to latch onto these successes, no matter how seemingly minor. We can’t yet make a pull request to staging until we uglify (that’s coming up), and we don’t want to make another web request to get this 2nd vendor javascript file. So we will hold this off in a separate branch until we complete those 2 items. We learned a bit about loaders and how they can be used to massage libraries to work with webpack. We also learned that we need to be careful with v1 vs v2 configuration options. Lastly, and most importantly, even when you think you are taking a small chunk (vendor), there’s always a smaller chunk you can work on (i.e. just Modernizr & Detectizr). Next time we’ll take a look at the rest of the vendors, and hopefully fully replace the vendor.min.js. If you made it this far, thank you for your time. You are part of a teeny tiny group, we meet on Tuesdays. Follow this journey from the beginning.

Journey to webpack

I decided this was something that needs to be written. Not much has been written on this topic and I know we’re not the only ones struggling with carrying a legacy application around while drooling over the latest shiny toys that we feel like we’ll never get to use. I don’t know how this is going to turn out. We may fail horribly, we may give up, we may get transferred to another project, but maybe, just maybe, there’s a chance we’ll succeed.

A lot has been written about the greatness that is webpack. However, not a lot has been written about transitioning a 3 year-old, 400k line AngularJS beast of mixed JavaScript and TypeScript bundled by Grunt to webpack.

A few weeks ago we were facing a few problems with our development environment:

  1. Adding a new “feature” required tweaking no less than 8 config files to satisfy the Grunt monster that runs about 30 tasks to create a build
  2. Running one of our 8500 unit tests with karma has a 30 second turn-arond time between saving a single file and getting a single test to run (using “fit”. If you ever interview with me, please know this answer).
  3. Our future, an Angular (4) application, was built recently from scratch to be more modularized, etc, etc. The original plan was that new modules could be created there, and downgraded via ng-upgrade to work within the AngularJS (v1) application. This worked great for a PoC and even a few features here and there, but the behemoth lingered. We needed a better upgrade plan, otherwise we’d never be able to switch off of the legacy code base. More on this later, as it’s very related, but more of an Angular topic than webpack.

At this point I wish to make a few related points:

  1. This site gets millions of uniques per day. It’s a great testament to the extraordinary team that built the foundation (before my time)
  2. We have a fairly solid CI environment which runs linters and unit tests multiple times a day on more than 20 branches
  3. I believe we’re facing a problem that many large applications face… More and more features get added while the infrastructure that supports it, is neglected and starts to bust at the seams. Management/Product doesn’t want to dedicate hours for something that “has always worked”. Someone must constantly push for keeping the environment sane and humming along. It is true, the application reflects the organization that built it.

False start

One of the first problems we identified in our build process is that it is “copy heavy”. We have about 5000 [J|T]S/[LE|C]SS/HTML files in our application. Grunt does a lot of work by copying these files around. Thankfully we have SSD locally, but our poor Jenkins build machines don’t, and it’s noticeable. Even locally though, the amount of processing and I/O is unnecessary and just slow enough to make it frustrating. We were able to make some headway by specifying an outDir for TS compilation and in the new Angular application, and we were able to use a symlinking method that is much faster in the new application. But that didn’t help the here and now.

So, a few of us discussed the idea of moving from Grunt to gulp so that we could take advantage of the streaming capabilities and not write to disk so often. The thinking was that if we just do all the file transformations in memory, it had to be faster. Unfortunately, the migration was a little bumpy, partially due to our inexperience with gulp, and I believe partially because it felt like we were replacing a hammer with a screwdriver, when we really wanted a power drill. This was also done “in our spare time” which normally isn’t a problem, but somehow the momentum was just never there.

gulp, like webpack, is great when you start from scratch, but can be a little mind bending when you are trying to replicate the (often crazy) gymnastics we made Grunt do to create a build. As we transitioned various tasks over, it became apparent that this was just a temporary band-aid and wouldn’t help the future of the application. I also believe it never gained traction because, while we were learning something new, gulp is no longer the new hotness, the transition just felt like grunt work (yeah I did). There is genuine excitement for learning to use webpack, especially as it grows in usage in the industry. I hate to admit it, but level-of-hotness does impact developer enthusiasm.

To give you an idea of what we’re up against, here are the Grunt tasks we run to perform a build: Names have been tweaked to protect the innocent.

‘clean:build’, ‘build-login-app’, ‘build-ng-upgrade’, ‘build-mobile-app’, ‘copy:production’, ‘lang-copy’, ‘es6-bundler’, ‘concat:production’, ‘preprocess:html’, ‘preprocess:app-js’, ‘uglify:app-js’, ‘uglify:vendor’, ‘uglify:app-components’, ‘less:production’, ‘sass’, ‘concat:vendorCss’, ‘postcss’, ‘cssmin’, ‘cssvalidator’, ‘strip_code’, ‘stripJsonComments’, ‘json-merge’, ‘json-minify’, ‘ngtemplates:app.framework.app.templates’, ‘ngtemplates:app.framework.ext.templates’, ‘ngtemplates:app.feature1.templates’, ‘ngtemplates:app.feature2.templates’, ‘ngtemplates:app.admin.templates’, ‘ngtemplates:app.feature3.templates’, ‘ngtemplates:app.feature4.templates’, ‘ngtemplates:app.feature5.templates’, ‘ngtemplates:app.feature6.templates’, ‘ngtemplates:app.feature7.templates’, ‘ngtemplates:app.feature8.templates’, ‘ngtemplates:app.feature9.templates’, ‘ngtemplates:app.settings.templates’, ‘ngtemplates:app.tour.templates’, ‘ngtemplates:app.feature10.templates’, ‘ngtemplates:app.feature11.templates’, ‘ngtemplates:app.feature12.templates’, ‘uglify:templates’, ‘htmlmin’, ‘replace:modules’, ‘replace:bust-cache’, ‘replace:app-js’, ‘replace:ga’, ‘replace:login’, ‘replace:ngUpgradeProd’, ‘clean:css’

Yup. That’s a lot. Each one of these calls several others. It’s crazy town.

What’s the best way to get some webpack going? Come back for part 2 where I’ll show you how we started taming the beast by bringing webpack in to perform very specific tasks.

[INSERT IMAGE OF DEVELOPMENT TEAM SLAYING A DRAGON WITH A webpack SWORD & SHIELD]