Archives: work

Journey to webpack, part 5 – misc

June 26, 2017 | By Brian | No Comments | Filed in: work.

New to the series? Start at the beginning

As we continue on our journey to convert Grunt to webpack, I realized I missed a few things that are probably useful.

  1. uglification
  2. source maps
  3. split dev & prod
  4. helper files
  5. ng-upgrade
  6. CommonChunksPlugin
  7. BundleAnalyzerPlugin

ugilfication

Uglification was one of the most straightforward config options we’ve added to webpack. Here’s what we use:

plugins: [
  new webpack.optimize.UglifyJsPlugin({
    /*eslint-disable camelcase*/
    beautify: false,
    output: {
      comments: false
    },
    mangle: {
      screw_ie8: true
    },
    compress: {
      drop_console: true,
      screw_ie8: true,
      warnings: false,
      conditionals: true,
      unused: true,
      comparisons: true,
      sequences: true,
      dead_code: true,
      evaluate: true,
      if_return: true,
      join_vars: true,
      negate_iife: false
   },
   sourceMap: true
   /*eslint-enable camelcase*/
 }))
]

Of note – the uglification of vendor adds 15 seconds to the build time (from 5sec to 20sec!). We also noticed that sourceMap: true adds ~2 seconds itself, but more importantly we couldn’t find the map files.

sourcemaps

Source maps are a must have. A little digging and we found the proper way to enable sourcemaps within webpack is via:

devtool: 'source-map',

Ah, much better ūüôā

split dev & prod

Given how much time uglification adds to the build, we don’t want to run it during development, we really just need it in our production build. We have 2 generally accepted methods of doing this: switching on NODE_ENV or 2 configs. We opted for the 2 configs because our config files were already getting big, and we didn’t want to add lots of if logic. It seemed like it would make it easier to debug.

So, off we go:

  1. Copy webpack.config.js to webpack-dev.config.js and webpack-prod.config.js
  2. Remove webpack.config.js
  3. Remove UglifyJsPlugin from the dev config
  4. Update the ‘webpack’ Grunt task
  5. Update references to the ‘webpack’ Grunt task
grunt.config('shell', {
  webpackDev: {
    command: 'node node_modules/webpack/bin/webpack --config webpack-dev.config.js --progress',
    options: {
      execOptions: { cwd: '.'},
      async: false, stdout: true, stderr: true, failOnError: true
    }
  },
  webpackProd: {
    command: 'node node_modules/webpack/bin/webpack --config webpack-prod.config.js --progress',
    options: {
      execOptions: { cwd: '.'},
      async: false, stdout: true, stderr: true, failOnError: true
    }
  },

This keeps our development build blazing fast!

Helper files

I’m not fan of duplication. Early on with all the work we did on the vendor libraries made our config file huge and it looked like we were doing the same things over & over. So we created vendor-helper.js. The immediate purpose of this file was to extract the repetitive expose-loader configurations, making our main config file more readable.

module.exports = {

  exposeLoaderConfig: [
    {
      name: 'lodash',
      global: '_'
    },
    {
      name: 'moment',
      global: 'moment'
    },
    {
      name: 'fastclick',
      global: 'FastClick'
    },
    {
      name: 'angular',
      global: 'angular'
    },
    {
      name: 'd3',
      global: 'd3'
    },
    {
      name: 'log4javascript',
      global: 'log4javascript'
    },
    {
      name: 'highcharts',
      global: 'Highcharts'
    }
  ],

  getSimpleGlobals: function() {
    return this.exposeLoaderConfig.map((config) => {
      return {
        test: require.resolve(config.name),
        use: [
          `expose-loader?${config.global}`
        ]
      };
    });
  }
};

Then in the main config file:

const VendorHelper = require('./webpack-config/vendor-helper');

module: {
  rules: {
    ...VendorHelper.getSimpleGlobals(),

 

Boom! We just made the config file that much smaller and more readable, getting those pesky libraries that need special handling out of our way. We don’t need to see that every day, so shoving them off to the side has brought a sense of calmness and clarity that we were missing… or something like that ūüôā

While we’re in the business of extracting stuff, the verbosity of the config for the¬†UglifyJsPlugin was bothersome, so we move it to a plugin-helper.ts. Again, this is something you essentially “set and forget”, and it takes up a lot of real estate in the editor.

Because of our pre-existing Grunt infrastructure, we have an environment variable called USE_MINIFIED, so we’ll piggy back on that to tell us whether or not we should uglify.

const webpack = require('webpack');

module.exports = {

  getUgilifyConfiguration: function() {
    let uglifyConfiguration = [];

    if (process.env.USE_MINIFIED === 'true') {

      uglifyConfiguration.push(
        new webpack.optimize.UglifyJsPlugin({
          /*eslint-disable camelcase*/
          beautify: false,
          output: {
            comments: false
          },
          mangle: {
            screw_ie8: true
          },
          compress: {
            drop_console: true,
            screw_ie8: true,
            warnings: false,
            conditionals: true,
            unused: true,
            comparisons: true,
            sequences: true,
            dead_code: true,
            evaluate: true,
            if_return: true,
            join_vars: true,
            negate_iife: false
          },
          sourceMap: true
          /*eslint-enable camelcase*/
        }))
    }

    return uglifyConfiguration;
  }
};

Then in our config file:

plugins: [
  ...PluginHelper.getUgilifyConfiguration(),

Can you tell we’re into the spread operator over here? ūüôā

You might be thinking… Hey I thought you moved to separate config files so you shouldn’t need to extract the uglify config AND have an ENV check, because it’s only in the prod config. Yeah, you are probably right. But we got the feeling that there were going to be more differences in the future…

ng-upgrade

I mentioned before that we have an Angular app we are trying to move our legacy AngularJS code base towards. Thus far we are using ng-upgrade to downgrade the Angular components to AngularJS successfully. To that end, we need to add these libraries to our webpack config. We have a Grunt task already in place to generate the ng-upgrade(able) bundle, we we’ll add that as an entry point.

entry: {
  'ng-upgrade/dist/bundle.js': './ng-upgrade/bootstrap-aot.ts',

We also need to tell the resolver where to look for these files, so we’ll add the following:

resolve: {
  modules: ['ng-upgrade', 'node_modules'],
},

The resolve.modules key¬†requires us to also include “node_modules”, which is there by default, but if you specify another path for modules, you have to include it.

THen, we need to include the AotPlugin.

plugins: [
  new ngToolsWebpack.AotPlugin({
    tsConfigPath: './ng-upgrade/tsconfig-aot.json'
  }),

Finally, we add the module.rules necessary to use the AotPlugin against our .ts files. Also, because our downgraded components contain style and templates, we need to tell webpack how to handle them. For this, we will simply use the raw-loader.

module: {
  rules: [
    {
      test: /\.ts$/,
      loader: '@ngtools/webpack'
    },
    { test: /\.css$/, loader: 'raw-loader' },
    { test: /\.html$/, loader: 'raw-loader' },

 

CommonChunksPlugin

The CommonChunksPlugin is one of those magical parts of webpack that “just works”. We’ll configure it to look for at least 2 common includes before it does the extraction into a separate JS file.

plugins: [
  new webpack.optimize.CommonsChunkPlugin({
      name: 'bundle',
      minChunks: 2,
      filename: 'common-chunks/[name].min.js'
  })

BundleAnalyzerPlugin

The BundleAnalyzerPlugin says that it generates a representation of ¬†“bundle content as convenient interactive zoomable treemap”. Simply put – it’s an amazing, interactive visual of the contents of the files webpack outputs.

Setup is really easy:

// eslint-disable-next-line no-unused-vars
const BundleAnalyzerPlugin = require('webpack-bundle-analyzer').BundleAnalyzerPlugin;
plugins: [
  new BundleAnalyzerPlugin(),

We put the eslint-disable line in there, because most of the time we don’t want to take the extra time to run it, but it’s nice to have the analyzer only a commented line away.

We then modify package.json to give us a quick shortcut we can execute with yarn or npm run

"inspect-common-chunk": "source-map-explorer build/static/app/common-chunks/bundle.min.js"
"inspect-ng-upgrade-bundle": "source-map-explorer build/static/app/ng-upgrade/dist/bundle.js",

Words cannot do this plugin justice… see for yourself:

BundleAnalyzer plugin

This is a great way to see how your config alters your output. And it’s fun to play with!

todo list

A quick checkin our our todo list shows that we did a bunch of stuff that wasn’t even on the list! This is the nature of working on something unknown, you never quite know what path you will take. It’s still important to have a list – so we all have a target.

  1. Create vendor.min.js
  2. Create app-components.min.js
  3. Create app.min.js
  4. Create templates.app.min.js
  5. Try out webpack 3.0!
  6. Figure out what’s up with moment.js
  7. Use the webpack’d files in karma for unit tests
  8. Try the CommonChunksPlugin
  9. Handle CSS/LESS/SASS
  10. Use the dev-server instead of grunt-contrib-connect

Journey to webpack, part 4 – the components

June 22, 2017 | By Brian | No Comments | Filed in: work.

New to the series? Start at the beginning

In part 3 we took care of our 3rd party libraries. Now, it’s time to deal with our 2nd party libraries.

What’s a 2nd party library?

Glad you asked – it’s the name I’ve given to libraries created by teams you collaborate with within your organization. This is usually where shared components and services go.

Aside: Sharing is always complicated. At what point does something get promoted to shared¬†status? Everyone likes to think that what they are building is going to be used by millions of people, or at least everyone in their department. But starting off building something in the shared space is tricky. Does it support everyone’s use cases? How long before it is used by more than one team? We’ve had shared implementations of stuff that was never even used! I try to push for a simple rule where nothing gets promoted into shared until the 2nd team needs it. This works pretty well most of the time, but has the downside of all teams needing to know what each other has built. Oh you have a Google Analytics service too? Yep, currently we have 5 of them. The bright side of this is that the requirements of each are well defined. Now we can take a look at all the functionality of each and try to boil it down into a single service. Easier said than done, of course.

But I digress…

We’ve got libraries for handling language, logging, buttons, modals, and pokerchips. Currently these files get uglified into app-components.min.js.

new entry point

So, let’s create a new entry point file that will process these and call it app-components.ts. It will will look like this:

 import '../node_modules/app-angular-logging/dist/js/app-angular-logging';
 import '../node_modules/app-angular-language/dist/js/app-angular-language';
 import '../node_modules/app-angular-utilities/dist/js/app-angular-utilities';
 import '../node_modules/app-angular-meta/dist/app-angular-meta';
 import '../node_modules/app-angular-ui-alert/dist/js/app-angular-ui-alert';
 import '../node_modules/app-angular-ui-button/dist/js/app-angular-ui-button';
 import '../node_modules/app-angular-ui-card/dist/js/app-angular-ui-card';
 import '../node_modules/app-angular-ui-containers/dist/js/app-angular-ui-containers';
 import '../node_modules/app-angular-ui-dashboard/dist/js/app-angular-ui-dashboard';
 import '../node_modules/app-angular-ui-form/dist/js/app-angular-ui-form';
 import '../node_modules/app-angular-ui-loading/dist/js/app-angular-ui-loading';
 import '../node_modules/app-angular-ui-pokerchip/dist/js/app-angular-ui-pokerchip';
 import '../node_modules/app-angular-ui-modal/dist/js/app-angular-ui-modal';
 import '../node_modules/app-angular-ui-popover/dist/js/app-angular-ui-popover';
 import '../node_modules/app-angular-ui-select/dist/js/app-angular-ui-select';
 import '../node_modules/app-angular-ui-tabs/dist/js/app-angular-ui-tabs';
 import '../node_modules/app-angular-ui-slidein/dist/js/app-angular-ui-slidein';
 import '../node_modules/app-shared/dist/app-shared';
 import '../node_modules/app-angular-time/dist/js/app-angular-time.shared';

Here’s what our config looks like now

output: {
  filename: 'build/static/app/vendor/js/vendor.min.js'
},
entry: {
  vendor: './webpack-config/vendor.ts'
},

We need to add a new entry point, but we only have one output filename. Luckily webpack comes with a variable substitution mechanism that uses¬†[name]. To take advantage of this, we’ll change entry.vendor to contain more of the destination, like this:

output: {
  filename: 'build/static/app/vendor/js/[name]'
},
entry: {
  'vendor.min.js': './webpack-config/vendor.ts',
},

Cool, that still work. Now we just need to add an entry for app-components.min.js

output: {
  filename: 'build/static/app/vendor/js/[name]'
},
entry: {
  'vendor.min.js': './webpack-config/vendor.ts',
  'app-components.min.js': './webpack-config/app-components.ts',
},

Errrr.. not so fast. Now app-components.min.js is being output into vendor/js/. Clearly not what we want.

Luckily, webpack has your back. It takes whatever you put as the key in the entry object as the [name] to substitute. Check this out:

output: {
  filename: 'build/static/app/[name]'
},

entry: {
  'vendor/js/vendor.min.js': './webpack-config/vendor.ts',
  'app-components.min.js': './webpack-config/app-components.ts',
},

YES! This results in exactly what we need:

 build/static/app/vendor/js/vendor.min.js
 build/static/app/app-components.min.js

Next, we can go and remove the ‘uglify:components’ from our list of Grunt tasks!

Now we can make a commit and relish in the glory of making our build process a slightly nicer place.

recap

Our homegrown components and libraries “just work” with webpack. Looking back, it would have been far less frustrating to start with these, but who knew?

Here is our updated webpack.config.js

Looking back at our TODO list, there is still much to do:

  1. Create vendor.min.js
  2. Create app-components.min.js
  3. Create app.min.js
  4. Create templates.app.min.js
  5. Try out webpack 3.0!
  6. Figure out what’s up with moment.js
  7. Use the webpack’d files in karma for unit tests
  8. Try the CommonChunksPlugin
  9. Handle CSS/LESS/SASS
  10. Use the dev-server instead of grunt-contrib-connect

Journey to webpack, part 3 – finish the vendors

June 19, 2017 | By Brian | No Comments | Filed in: work.

Welcome to part 3 of our journey, catch up with part 1 and part 2.

quick recap

We’ve injected webpack into our build process, it’s run before ‘uglify:vendor’ and is bundling 2 of our 27 3rd party libraries (detectizr & modernizr). We did this by paring down our vendor.ts file, which we use as an entry point, then massaging the libraries to work with webpack, using various loaders. Currently we have 2 “vendor” files that are mutually exclusive.

{
  test: require.resolve('modernizr'),
  use: [
   'expose-loader?Modernizr',
   'imports-loader?this=>window',
   'exports-loader?window.Modernizr'
  ]
},
{
  test: require.resolve('detectizr'),
  use: [
   'expose-loader?Detectizr',
   'imports-loader?this=>window',
   'exports-loader?window.Detectizr'
  ]
},

The next step is to get the rest of the vendor files converted over to use webpack, and remove the ‘uglify:vendor’ task completely.

let’s get to it

Let’s take it from the top. Here is what our vendor.ts file looks like:

 import '../node_modules/modernizr/modernizr';
 import '../node_modules/detectizr/dist/detectizr';
 // import '../node_modules/jquery/dist/jquery';
 // import '../node_modules/jquery.easing/jquery.easing';
 // import '../node_modules/lodash/index';
 // import '../node_modules/moment/min/moment-with-locales.min';
 // import '../node_modules/moment-timezone/builds/moment-timezone-with-data';
 // import '../node_modules/d3/d3';
 // import '../node_modules/bootstrap/dist/js/bootstrap';
 // import '../node_modules/fastclick/lib/fastclick';
 // import '../node_modules/angular/angular';
 // import '../node_modules/angular-cookie/angular-cookie';
 // import '../node_modules/angular-cookies/angular-cookies';
 // import '../node_modules/angular-ui-bootstrap/ui-bootstrap-tpls';
 // import '../node_modules/angular-moment/angular-moment';
 // import '../node_modules/angular-resource/angular-resource';
 // import '../node_modules/angular-sanitize/angular-sanitize';
 // import '../node_modules/angular-animate/angular-animate';
 // import '../node_modules/angular-touch/angular-touch';
 // import '../node_modules/angular-ui-router/release/angular-ui-router';
 // import '../node_modules/angulartics/src/angulartics';
 // import '../node_modules/angulartics/src/angulartics-ga';
 // import '../node_modules/log4javascript/log4javascript';
 // import '../node_modules/isotope/dist/isotope.pkgd';
 // import '../node_modules/phoneformat/phoneformat.min';
 // import '../node_modules/jquery-form/jquery.form';
 // import '../node_modules/intl-tel-input/build/js/intlTelInput';
 // import '../node_modules/intl-tel-input/lib/libphonenumber/build/utils';
 // import '../node_modules/bootstrap-colorpickersliders/dist/tinycolor';
 // import '../node_modules/bootstrap-colorpickersliders/dist/bootstrap.colorpickersliders';
 // import '../node_modules/ng-sortable/dist/ng-sortable';
 // import '../node_modules/highcharts/lib/highcharts';
 // import '../node_modules/highcharts/lib/modules/drilldown';
 // import '../node_modules/highcharts/lib/modules/exporting';
 // import '../node_modules/highcharts/lib/modules/heatmap';
 // import '../node_modules/oclazyload/dist/ocLazyLoad';
 // import '../node_modules/tslib/tslib';

We’ll go down the list one-by-one, uncommenting from here, and commenting out from our ‘uglify:vendor’ task then running the server to make sure there are no console errors in the browser.

First up: jQuery

Uncaught ReferenceError: jQuery is not defined

Looks like something in vendor.min.js (the non-webpack’d file) expects jQuery to already be loaded and to be defined on window. Luckily, we know how to fix this via the expose-loader. Since jQuery is also often referred to as $, we need to expose it as $ too.

{
  test: require.resolve('jquery'),
  use: [
   'expose-loader?jQuery',
   'expose-loader?$'
  ]
},

Success!

fast-forward a bit

We can clearly see where this is headed… let’s use expose-loader for all the libraries that create global variables: lodash, moment, fastclick, angular, d3, log4javascript, and highcharts

{
  test: require.resolve('lodash'),
  use: [ 'expose-loader?_' ]
},
{
  test: require.resolve('moment'),
  use: [ 'expose-loader?moment' ]
},
{
  test: require.resolve('fastclick'),
  use: [ 'expose-loader?FastClick' ]
},
{
  test: require.resolve('angular'),
  use: [ 'expose-loader?angular' ]
},
{
  test: require.resolve('d3'),
  use: [ 'expose-loader?d3' ]
},
{
  test: require.resolve('log4javascript'),
  use: [ 'expose-loader?log4javascript' ]
},
{
  test: require.resolve('highcharts'),
  use: [ 'expose-loader?Highcharts' ]
},

Seems simple enough.

But whooops, something is wrong with moment.

WARNING in ./~/moment/min/moment-with-locales.min.js
Module not found: Error: Can't resolve './locale' in '/Users/oloreb/dev/app/node_modules/moment/min'
@ ./~/moment/min/moment-with-locales.min.js 1:2798-2820
@ ./webpack-config/vendor.ts

A quick Google search and people suggesting the IgnorePlugin. Apparently the pathing for moment’s locale files is funky, so we will just ignore them for now. We’ll queue it up to deal with later.

new webpack.IgnorePlugin(/\.\/locale$/)

Success with no warnings this time! OK, what’s left now?

 // import '../node_modules/bootstrap/dist/js/bootstrap';
 // import '../node_modules/angular-cookie/angular-cookie';
 // import '../node_modules/angular-cookies/angular-cookies';
 // import '../node_modules/angular-ui-bootstrap/ui-bootstrap-tpls';
 // import '../node_modules/angular-moment/angular-moment';
 // import '../node_modules/angular-resource/angular-resource';
 // import '../node_modules/angular-sanitize/angular-sanitize';
 // import '../node_modules/angular-animate/angular-animate';
 // import '../node_modules/angular-touch/angular-touch';
 // import '../node_modules/angular-ui-router/release/angular-ui-router';
 // import '../node_modules/angulartics/src/angulartics';
 // import '../node_modules/angulartics/src/angulartics-ga';
 // import '../node_modules/isotope/dist/isotope.pkgd';
 // import '../node_modules/phoneformat/phoneformat.min';
 // import '../node_modules/jquery-form/jquery.form';
 // import '../node_modules/intl-tel-input/build/js/intlTelInput';
 // import '../node_modules/intl-tel-input/lib/libphonenumber/build/utils';
 // import '../node_modules/bootstrap-colorpickersliders/dist/tinycolor';
 // import '../node_modules/bootstrap-colorpickersliders/dist/bootstrap.colorpickersliders';
 // import '../node_modules/ng-sortable/dist/ng-sortable';
 // import '../node_modules/oclazyload/dist/ocLazyLoad';
 // import '../node_modules/tslib/tslib';

Not much! Next we knock out all the angular stuff, bootstrap, ng-sortable, ocLazyLoad and tslib.

This is going smooth!

Leaving us with a few odd balls:

 // import '../node_modules/isotope/dist/isotope.pkgd';
 // import '../node_modules/phoneformat/phoneformat.min';
 // import '../node_modules/intl-tel-input/build/js/intlTelInput';
 // import '../node_modules/intl-tel-input/lib/libphonenumber/build/utils';

Uncommenting isotope, leaves us with an error which we can resolve with our old friend imports-loader

{
  test: require.resolve('isotope/dist/isotope.pkgd'),
  use: [
   'imports-loader?this=>window'
  ]
}

Note: this is only a problem with isotope v2. If you are using isotope v3, use this method.

the final three

The last 3 items all deal with our telephone number input boxes.
Uncommenting phoneformat leaves us with:

i18n is not defined

Uncommenting intl-tel-input leaves us with with the same thing.

Neither just work on their own, so let’s dig into phoneformat

phoneformat uses a Google module loading system, so first we need to use imports-loader to convert “this=>window”

It’s also creating a global variable called goog, so we need to expose it via the expose-loader.
Then, similar to modernizr, we also need the exports-loader to export the goog variable.

{
  test: require.resolve('phoneformat'),
  use: [
   'expose-loader?goog',
   'imports-loader?this=>window',
   'exports-loader?goog'
  ]
},

This isn’t so bad, but we’re still not working. Same error, but at least window.goog is available.

We have libraries that expecting window.i18n to be available, but it’s different than window.goog, and we can only use expose-loader to expose a single variable (or, as with jQuery, expose a single variable with different names: jQuery & $).

Looking into the source of phoneformat, we find a a line declaring the i18n variable.  Not to be confused with window.goog.i18n, which is completely different (but threw me off the track for a solid hour!)

So, I thought to myself, hey if that said “window.i18n =” instead of “var i18n =”, we’d be in business. Luckily there is a plugin that lets you do string replacement:

{
  test: require.resolve('phoneformat'),
  use: [
   'expose-loader?goog',
   'imports-loader?this=>window',
   'exports-loader?goog',
   {
     loader: StringReplacePlugin.replace({
     replacements: [
       {
         pattern: /var i18n/ig,
         replacement: function (match, p1, offset, string) {
           return 'window.i18n';
         }
       }
     ]})
   }
  ]
},

Bingo! Nailed it! While this feels incredibly hacky, it’s 100% functional and the logic checks out. The string replacement is confined to the phoneformat JS file, and won’t affect others. We’ll have to revisit if we ever upgrade the library, but we can worry about that later. * For the record, even though it sounds like this took minutes, it took a few hours over the course of a few days to come up with this solution. In hindsight, it seems obvious, but reality is that this one had us stumped for a while. It’s good to know that webpack has your back for situations like these.

Now we switch over intl-tel-input, and it works without further changes!

Now we can go back through and remove references to files and tasks that build the old vendor.min.js (remember we’re using vendor-webpack.min.js). This includes the the grunt tasks and index.html. In fact, we’ll leave the references to vendor.min.js in the index.html file, and switch our webpack config to create vendor.min.js.

That’s it! We have now replaced ‘uglify:vendor’ with ‘shell:webpack’ to generate vendor.min.js!

Looking ahead, we’ve still got lots more for webpack to do:

  1. Create app-components.min.js
  2. Create templates.app.min.js
  3. Create app.min.js
  4. Handle CSS/LESS/SASS
  5. Try out webpack 3.0!
  6. Use the webpack’d files in karma for unit tests
  7. Use the dev server instead of grunt-contrib-connect
  8. Try the CommonChunksPlugin
  9. Figure out what’s up with moment.js

for reference

Here’s our webpack.config.js

Here are the versions of these libraries that we are currently using. Things change, and maybe newer releases of these libraries will play more nicely with webpack. So, here are the relevant entries from package.json:

 "angular": "1.5.5",
 "angular-cookie": "4.0.10",
 "angular-cookies": "1.5.5",
 "angular-moment": "0.10.3",
 "angular-resource": "1.5.5",
 "angular-sanitize": "1.5.5",
 "angular-touch": "1.5.5",
 "angular-resource": "1.5.5",
 "angular-sanitize": "1.5.5",
 "angular-touch": "1.5.5",
 "bootstrap-colorpickersliders": "3.0.2",
 "bootstrap": "3.3.5",
 "d3": "3.5.6",
 "detectizr": "2.0.0",
 "fastclick": "1.0.6",
 "highcharts": "4.1.10",
 "intl-tel-input": "3.6.1",
 "isotope": "2.0.0",
 "jquery": "2.1.4",
 "jquery-form": "3.46.0",
 "jquery.easing": "1.3.2",
 "lodash": "3.10.1",
 "log4javascript": "1.4.15",
 "modernizr": "2.8.3",
 "moment": "2.10.6",
 "moment-timezone": "0.5.1",
 "ng-sortable": "1.3.0",
 "oclazyload": "1.0.6",
 "phoneformat": "0.0.7",
 "tslib": "1.5.0",

Journey to webpack, part 2 – One small step

June 14, 2017 | By Brian | 1 Comment | Filed in: work.

If you missed the first part of this journey, pop over to Part 1.

OK, let’s get rolling here and jump right in…. let’s convert everything to use webpack and reap the benefits!

YES!

Me: “Let’s start easy…. OK, how do I concat then¬†minify then…”
Me: 10 minutes later: “Oh shit, there is no thenmind blown

OK, so webpack is a completely different mindset to the task-runner-approach I am familiar with like Make/Rake/Grunt/Gulp, heck even Ant. Different doesn’t mean bad (that’s what I tell my kids!), but this isn’t going to be a quick transition.

So where do we start, for real?

We’ve had Grunt tasks in place for years (literally) so we already create minified bundles. These are the main ones:

 <script src="vendor/js/vendor.min.js"></script>
 <script src="app-components.min.js"></script>
 <script src="templates.app.min.js"></script>
 <script src="app.min.js"></script>

We decided that the first step should be to attack our 3rd party dependencies – namely anything in node_modules/ that we didn’t create. We still call it “vendor” because we used to use Bower for everything. Then we used a mix of Bower and npm, and finally now we are 100% npm (well some of us moved to yarn, but npm v5 looks sweet… I digress). But the name “vendor” has stuck, and I think that’s OK.

Aside: Bower is a good reminder that all software has a shelf life. Nothing lasts forever in this crazy fast changing world of software. I don’t know what will replace webpack, but we’re being diligent about our inputs and outputs, and we should have an easier time transitioning to the next thing because we are more educated on just what exactly is in our project. Also, in a weird way, the webpack config seems kinda self documenting, which is awesome.

We settled on vendor.min.js because:

  1. It’s all 3rd party stuff that¬†has hopefully already been webpack’d by someone
  2. It’s all JavaScript (no TypeScript to worry about… yet)
  3. We could literally replace the “uglify:vendor” task with a new task named “shell:webpack”

By approaching it this way, we could immediately merge into staging (our production line) without needing to wait until we have the entire project converted. We were fearful this could take weeks/months to fully complete and wanted to show progress… and also feel good about ourselves.

baby steps

webpackify the vendors

If you’re like us, you have more 3rd party dependencies than you’d like to admit. It’s cool. We all do it. We’re trying to be more stringent now, but back when we started, we’d pull in anything and everything. Hence we have 37 distinct JavaScript files we rely on.

Yup. 37.

Stop me if this sounds familiar: modernizr, detectizr, jQuery, lodash, moment, d3, bootstrap, fastclick, angular, angular-cookie, angular-cookies (wha?), angular-animate, angular-ui-router, angular-moment, angulartics, isotope, phoneformat, bootstrap-colorpickers, highcharts, ng-sortable, ocLazyLoad, log4javascript… you get the idea.

Honestly at this point it doesn’t matter if it’s 2 or 22, we have a bunch of files from other places that we need to concat & minify into a single file named vendor.min.js. This isn’t rocket science. Let’s see how we’d do it in webpack.

here we go

First, we specify where we want this file to go. That looks easy enough: there is a config option named “output” that can take a “filename”. We know where we want it to end up and what to call it so:

module.exports = {
  output: {
    filename: 'build/static/app/vendor/js/vendor.min.js'
   },
 }

Now, how do we tell webpack to “go grab all these things in node_modules/ and do your thing”?

webpack has this concept of an “entry point”. Sweet! I can just point it at the entry point of my application and … DOH! Wait a sec. I can’t do that. If I do that, it’s going to try to do everything, I want to do this systematically and only work on node_modules/ … Oh and did I mention that I don’t have an actual entry point? This is a Angular 1.5 application built with a mix of JavaScript and TypeScript. Shame on me. I thought our entry point might be index.html!

Clearly pointing webpack at node_modules/ wasn’t going to work (yeah that was my first hope). Even if somehow it did work, we’d have to filter out anything that we’ve written ourselves (what I call 2nd party libraries), plus all the junk that isn’t used at runtime… yeah it’d be a mess.

create your own entry point

It’s fun¬†working with smart people. This time we leaned on Tor, who has a little webpack experience. He’s getting credit for this whether he figured it out¬†himself, read it somewhere or just took a wild guess (I’m betting on the first one). ¬†Say hello to the “entry point file”. Basically what we will do is create a new file that imports all the files that we want in the bundle. In other words, we’re faking it ’til we’re making it…. and Tor approves.

Tor to the rescue!

So we build up a file called vendor.ts that looks like this:

 import '../node_modules/modernizr/modernizr';
 import '../node_modules/detectizr/dist/detectizr';
 import '../node_modules/jquery/dist/jquery';
 import '../node_modules/jquery.easing/jquery.easing';
 import '../node_modules/lodash/index';
 import '../node_modules/moment/min/moment-with-locales.min';
 import '../node_modules/moment-timezone/builds/moment-timezone-with-data';
 import '../node_modules/d3/d3';
 import '../node_modules/bootstrap/dist/js/bootstrap';
 import '../node_modules/fastclick/lib/fastclick';
 ...
 ...

Then we update our webpack.config.js to use it as an entry point:

module.exports = {
  output: {
    filename: 'build/static/app/vendor/js/vendor/min.js'
  },  
  entry: {
    vendor: './webpack-config/vendor.ts',
  },
}

Let’s do this! At this point, we’ve created our webpack Grunt task, so we can just run “grunt shell:webpack”…

 Running "shell:webpack" (shell) task
 Hash: 4580ce86f289c11a996b
 Version: webpack 2.6.1
 Time: 4245ms
 Asset Size Chunks Chunk Names
 build/static/redbox/vendor/js/vendor.min.js 5.15 MB 0 [emitted] [big] vendor
 build/static/redbox/vendor/js/vendor.min.js.map 6.25 MB 0 [emitted] vendor
 [1] ./~/jquery/dist/jquery.js 248 kB {0} [built]
 [112] ./~/intl-tel-input/lib/libphonenumber/build/utils.js 217 kB {0} [built]
 [113] ./~/isotope/dist/isotope.pkgd.js 103 kB {0} [built]
 [114] ./~/jquery-form/jquery.form.js 42.5 kB {0} [built]
 [115] ./~/jquery.easing/jquery.easing.js 4.87 kB {0} [built]
 [116] ./~/lodash/index.js 411 kB {0} [built]
 [117] ./~/log4javascript/log4javascript.js 128 kB {0} [built]
 [118] ./~/modernizr/modernizr.js 51.4 kB {0} [built]
 [119] ./~/moment-timezone/builds/moment-timezone-with-data.js 193 kB {0} [built]
 [120] ./~/moment/min/moment-with-locales.min.js 150 kB {0} [built]
 [121] ./~/ng-sortable/dist/ng-sortable.js 38.9 kB {0} [built]
 [122] ./~/oclazyload/dist/ocLazyLoad.js 57.7 kB {0} [built]
 [123] ./~/phoneformat/phoneformat.min.js 433 kB {0} [built]
 [124] ./~/tslib/tslib.js 6.47 kB {0} [built]
 [127] ./webpack-config/vendor.ts 2.08 kB {0} [built]
 + 113 hidden modules

SUCCESS! We’ve got a vendor.min.js that weighs in at a hefty 5MB (yup, this is before we uglify… patience my friend), it’s in the right place, and now we should be able to start up our application!

make it part of the build

Next we replace the Grunt task “uglify:vendor” with “shell:webpack” and cross our fingers while our dev server starts up.

Fire up the browser, and …

vendor.min.js:108094 Uncaught TypeError: Cannot read property ‘documentElement’ of undefined
index.html:35 Uncaught ReferenceError: angular is not defined

That’s weird. After a quick double check, angular.js is in fact in our¬†entry point file and, because we didn’t yet uglify, I can actually see it in vendor.min.js.

Not sure what to do next, we decided¬†– “OK if they don’t ALL work, let’s try paring them down”. This required putting back the “uglify:vendor” task and commenting out only the inclusion of modernizr & detectizr. Back in vendor.ts, we remove everything but modernizr & detectizr. We also had to modify our setup slightly. We’ll have Grunt continue to build vendor.min.js and we will have webpack build a new file called vendor-webpack.min.js. Finally, we’ll modify index.html to load vendor-webpack.min.js before vendor.min.js. PHEW!

 Hash: a76a5e5b36fcdc8a8181
 Version: webpack 2.6.1
 Time: 187ms
 Asset Size Chunks Chunk Names
 build/static/redbox/vendor/js/vendor-webpack.min.js 73.7 kB 0 [emitted] vendor
 build/static/redbox/vendor/js/vendor-webpack.min.js.map 87.4 kB 0 [emitted] vendor
 [0] ./~/detectizr/dist/detectizr.js 16.8 kB {0} [built]
 [1] ./~/modernizr/modernizr.js 51.4 kB {0} [built]
 [2] ./webpack-config/vendor.ts 2.08 kB {0} [built]

Fire up the browser, and …

Uncaught TypeError: Cannot read property ‘detect’ of undefined

What now? I think this is better, but WTF?

to the google

Having really¬†no clue what’s going on, we took to Google and quickly learned that we need to let webpack know about any libraries that expose global variables. This is done with the expose-loader.

OK, this isn’t so bad:

yarn add expose-loader

Since both modernizr and detectizr expose globals, we’ll add both to webpack.config.js:

module: {
  rules: [
    {
      test: require.resolve('modernizr'),
      use: [{
        loader: 'expose-loader',
        options: 'Modernizr'
      }]
    },
    {
      test: require.resolve('detectizr'),
      use: [{
        loader: 'expose-loader',
        options: 'Detectizr'
      }]
    }
  ]
}

Fire up the browser, and …

Uncaught TypeError: Cannot read property ‘documentElement’ of undefined

Hmmm. That looks familiar. It’s the same error we had before, but it seems that window.Modernizr is defined at least. And we don’t see the error about Detectizr.detect. In the devtools console we can see that both Modernizr and Detectizr are globally available. We’ll call this progress!?

fixing the ‘izrs

Again, not really having any clue about where to go next, we check Google and find some links specific to modernizr. As much as I hate the copy-paste-pray method, sometimes it’s all you got.

yarn add imports-loader exports-loader

So we blindly add to our configuration:

module: {
  rules: [
    {
      test: require.resolve('modernizr'),
      use: [{
        loader: 'expose-loader',
        options: 'Modernizr'
      }, {
        loader: 'imports-loader?this=>window!exports-loader?window.Modernizr'
      }]
    },
    {
      test: require.resolve('detectizr'),
      use: [{
        loader: 'expose-loader',
        options: 'Detectizr'
      }, {
        loader: 'imports-loader?this=>window!exports-loader?window.Detectizr'
      }]
    }
  ]
}

Fire up the browser, and …

No errors! It works! Commit! Push! Call it a day!

Well not so quick. What the heck is this thing doing?

the import/export business

Here we learn that many NodeJS modules are written to expect “this” to be the global space, and browser’s call the global space “window”. So what the imports-loader does is transform “this” to “window” with the “this=>window” syntax.¬†Read more about shimming.

Let’s break this down a bit:

loader: 'imports-loader?this=>window!exports-loader?window.Modernizr'
         |------------|||----------|||------------|||--------------|
               |       |      |     |      |       |       |
               |      here    |  loader    |     here      |
               |   come some  | separator  |   come some   |
               |   parameters |            |   parameters  |
               |              |            |               |
             loader          the         loader           the
              name          params        name           params
                         to the loader                to the loader

Aside: This is probably the last time I try to do an ascii diagram by hand. If you are on a phone, I am sorry, it’s kind of a mess. Admittedly it was fun to make. By hand. With vim. Yup.

This little shorthand is intimidating at first, but becomes more clear when you use it the second time. Now, if you’re paying attention, it seems we are both “exporting” and “exposing” window.Modernizr. What’s up with that? Do we need both?

Well, it certainly works with both, let’s try without expose-loader

Running webpack results in this error:

ERROR in ./~/modernizr/modernizr.js
 Module parse failed: /Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/imports-loader/index.js?this=>window!exports-loader?window.Modernizr!/Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/modernizr/modernizr.js Unexpected token (1411:13)
 You may need an appropriate loader to handle this file type.
 | })(this, this.document);
 |
 | }.call(window!exports-loader?window.Modernizr));
 @ ./webpack-config/vendor.ts 1:0-45

ERROR in ./~/detectizr/dist/detectizr.js
 Module parse failed: /Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/imports-loader/index.js?this=>window!exports-loader?window.Detectizr!/Users/oloreb/dev/redbox-pay/redbox_angular/node_modules/detectizr/dist/detectizr.js Unexpected token (507:13)
 You may need an appropriate loader to handle this file type.
 | }(this, this.navigator, this.document));
 |
 | }.call(window!exports-loader?window.Detectizr));
 @ ./webpack-config/vendor.ts 2:0-50

OK, so it’s needed to even run webpack, but it’s not entirely clear why.

Well, let’s try to consolidate these loaders. The shorthand syntax is pretty sweet, so let’s append the expose-loader and just make this a one-liner:

loader: 'imports-loader?this=>window!exports-loader?window.Modernizr!expose-loader?Modernizr'

Ugh. That didn’t work. We get the same build error as above.

According to the documentation, webpack v2 doesn’t support chaining this way, so maybe we shouldn’t head down this road. Let’s back up a bit and clearly define the 3 things¬†we know need to happen:

      { 
        test: require.resolve('modernizr'),
        use: [{
          loader: 'expose-loader',
          options: 'Modernizr'
        }, {
          loader: 'imports-loader',
          options: 'this=>window'
        }, {
          loader: 'exports-loader',
          options: 'window.Modernizr'
        }]
      },

Ordering here is important. webpack processes this array bottom->top. We need the exports-loader to run before imports-loader.

I still haven’t figured out why we need both the expose-loader and the exports-loader, but we’re back to working again. It seems it would only be necessary if you were using an import or require. Maybe it’s because we’re importing it in vendor.ts? I figured if we exposed it, that would be enough, because it seems all that is needed is for the variable to be available globally. (TODO: Update when I figure out why)

Now let’s see if we can pass the “options” as parameters to the loader like in webpack v1

      {
        test: require.resolve('modernizr'),
        use: [{
          loader: 'expose-loader?Modernizr'
        }, {
          loader: 'imports-loader?this=>window'
        }, {
          loader: 'exports-loader?window.Modernizr'
        }]
      },

As expected, this also works. And by the way, I am loving the short-hand!

clean up in aisle 2

If we go back & look at the documentation, webpack v2 will also accept an array of loaders in the “use” option. So we can make this even more pleasing to the eye¬†with:

   {
      test: require.resolve('modernizr'),
      use: [
        'expose-loader?Modernizr',
        'imports-loader?this=>window',
        'exports-loader?window.Modernizr'
      ]
    },
    {
      test: require.resolve('detectizr'),
      use: [
        'expose-loader?Detectizr',
        'imports-loader?this=>window',
        'exports-loader?window.Detectizr'
      ]
    }

Now we’re talking!

git commit -am "webpack: vendor-webpack.min.js contains modernizr and detectizr" && git push origin HEAD

one giant leap

We’ll end here for today. We need to latch onto these successes, no matter how seemingly minor. We can’t yet¬†make a pull request to staging until we uglify (that’s coming up), and we don’t want to make another web request to get this 2nd vendor javascript file. So we will hold this off in a separate branch until we complete those 2 items.

We learned a bit about loaders and how they can be used to massage libraries to work with¬†webpack. We also learned that we need to be careful with v1 vs v2 configuration options. Lastly, and most importantly, even when you think you are taking a small chunk (vendor), there’s always a smaller chunk you can work on (i.e. just Modernizr & Detectizr).

Next time we’ll take a look at the rest of the vendors, and hopefully fully replace the vendor.min.js.

If you made it this far, thank you for your time. You are part of a teeny tiny group, we meet on Tuesdays.

Follow this journey from the beginning.

Journey to webpack

June 9, 2017 | By Brian | 4 Comments | Filed in: angular, work.

I decided this was something that needs to be written. Not much has been written on this topic and I know we’re not the only ones struggling with carrying a legacy application around while drooling over the latest shiny toys that we feel like we’ll never get to use. I don’t know how this is going to turn out. We may fail horribly, we may give up, we may get transferred to another project, but maybe, just maybe, there’s a chance we’ll succeed.

A lot has been written about the greatness that is webpack. However, not a lot has been written about transitioning a 3 year-old, 400k line AngularJS beast of mixed JavaScript and TypeScript bundled by Grunt to webpack.

A few weeks ago we were facing a few problems with our development environment:

  1. Adding a new “feature” required tweaking no less than 8 config files to satisfy the Grunt monster that runs about 30 tasks to create a build
  2. Running one of our 8500 unit tests with karma has a 30 second turn-arond time between saving a single file and getting a single test to run (using “fit”. If you ever interview with me, please know this answer).
  3. Our future, an Angular (4) application, was built recently from scratch to be more modularized, etc, etc. The original plan was that new modules could be created there, and downgraded via ng-upgrade to work within the AngularJS (v1) application. This worked great for a PoC and even a few features here and there, but the behemoth lingered. We needed a better upgrade plan, otherwise we’d never be able to switch off of the legacy code base. More on this later, as it’s very related, but more of an Angular topic than webpack.

At this point I wish to make a few related points:

  1. This site¬†gets millions of uniques per day. It’s a great testament to the extraordinary team that built the foundation (before my time)
  2. We have a fairly solid CI environment which runs linters and unit tests multiple times a day on more than 20 branches
  3. I believe we’re facing a problem that many large applications face… More and more features get added while the infrastructure that supports it, is neglected and starts to bust at the seams. Management/Product doesn’t want to dedicate hours for something that “has always worked”. Someone must constantly push for keeping the environment sane and humming along. It is true, the application reflects the organization that built it.

False start

One of the first problems we identified in our build process is that it is “copy heavy”. We have about 5000 [J|T]S/[LE|C]SS/HTML files in our application. Grunt does a lot of work by copying these files around. Thankfully we have SSD locally, but our poor Jenkins build machines don’t, and it’s noticeable. Even locally though, the amount of processing and I/O is unnecessary and just slow enough to make it frustrating. We were able to make some headway by specifying an outDir for TS compilation and in the new Angular application, and we were able to use a symlinking method that is much faster in the new application. But that didn’t help the here and now.

So, a few of us discussed the idea of moving from Grunt to gulp so that we could take advantage of the streaming capabilities and not write to disk so often. The thinking was that if we just do all the file transformations in memory, it had¬†to be faster. Unfortunately, the migration was a little bumpy, partially due to our inexperience with gulp, and I believe partially because it felt like we were replacing a hammer with a screwdriver, when we really wanted a power drill. This was also done “in our spare time” which normally isn’t a problem, but somehow the momentum was just never there.

gulp, like webpack, is great when you start from scratch, but can be a little mind bending when you are trying to replicate the (often crazy) gymnastics we made Grunt do to create a build. As we transitioned various tasks over, it became apparent that this was just a temporary band-aid and wouldn’t help the future of the application. I also believe it never gained traction because, while we were learning something new, gulp is no longer the new hotness, the transition just felt like grunt work (yeah I did). There is genuine excitement for learning to use webpack, especially as it grows in usage in the industry. I hate to admit it, but level-of-hotness does impact developer enthusiasm.

To give you an idea of what we’re up against, here are the Grunt tasks we run to perform a build:
Names have been tweaked to protect the innocent.

‘clean:build’,
‘build-login-app’,
‘build-ng-upgrade’,
‘build-mobile-app’,
‘copy:production’,
‘lang-copy’,
‘es6-bundler’,
‘concat:production’,
‘preprocess:html’,
‘preprocess:app-js’,
‘uglify:app-js’,
‘uglify:vendor’,
‘uglify:app-components’,
‘less:production’,
‘sass’,
‘concat:vendorCss’,
‘postcss’,
‘cssmin’,
‘cssvalidator’,
‘strip_code’,
‘stripJsonComments’,
‘json-merge’,
‘json-minify’,
‘ngtemplates:app.framework.app.templates’,
‘ngtemplates:app.framework.ext.templates’,
‘ngtemplates:app.feature1.templates’,
‘ngtemplates:app.feature2.templates’,
‘ngtemplates:app.admin.templates’,
‘ngtemplates:app.feature3.templates’,
‘ngtemplates:app.feature4.templates’,
‘ngtemplates:app.feature5.templates’,
‘ngtemplates:app.feature6.templates’,
‘ngtemplates:app.feature7.templates’,
‘ngtemplates:app.feature8.templates’,
‘ngtemplates:app.feature9.templates’,
‘ngtemplates:app.settings.templates’,
‘ngtemplates:app.tour.templates’,
‘ngtemplates:app.feature10.templates’,
‘ngtemplates:app.feature11.templates’,
‘ngtemplates:app.feature12.templates’,
‘uglify:templates’,
‘htmlmin’,
‘replace:modules’,
‘replace:bust-cache’,
‘replace:app-js’,
‘replace:ga’,
‘replace:login’,
‘replace:ngUpgradeProd’,
‘clean:css’

Yup. That’s a lot. Each one of these calls several others. It’s crazy town.

What’s the best way to get some webpack going? Come back for part 2 where I’ll show you how¬†we¬†started taming the beast¬†by bringing webpack in to perform very specific tasks.

[INSERT IMAGE OF DEVELOPMENT TEAM SLAYING A DRAGON WITH A webpack SWORD & SHIELD]