kdz – My Personal Scaffolding Tool

I came up with some cool techniques after working on a bunch of web development projects, techniques based mostly around front-end tooling. I knew I would use these techniques and tools in future projects, and also knew that I had to find a way to easily set them up at the start of each project.

I solved the problem by creating a Node scaffolding CLI tool. I’m also being cute about it and naming it based on my hacker alias, calling it “kdz”.

Table of Contents

  1. Some notes
  2. How This Started
  3. The Problem
  4. More Problems
  5. The Tool-Building Process
  6. The Final Command
  7. Behind The Scenes
  8. What’s Left To Do
  9. Further Reading
  10. Conclusion

Some Notes

A few notes first…

How This Started

I had a project at work where I had to create a single page website using a standard HTML5/CSS/JavaScript web stack. The project tooling was set up like this in my dev environment:

  • Jade was used for HTML5 pre-processing.
  • LESS was used for CSS pre-processing.
  • CoffeeScript was used for JavaScript pre-processing.
  • I used the core CSS file that comes with Twitter Bootstrap because it I’m very productive when using it to build a responsive web design.
  • Bower was used to manage browser-level dependencies via a bower.json file.
  • npm was used to manage development-level dependencies via a package.json file.
  • Grunt and Gulp were used to automate various project tasks, especially build-out of the final page.

I had hit a serious comfort zone with the tools well in advance of starting this project. I had used all of them before and knew how to get them to work as a team inside my dev environment.

This was especially true of the CSS processing, which was powered by Gulp…mostly. Gulp let me create an efficient process for generating a single, production-ready CSS file…linting it, minifying it, concatenating it, etc.

I actually hit a point where I felt compelled to tweet this…

There was a rush to launch this project and as a result, it launched when the code was “good enough,” but “not as good as it could’ve been.” Some things clearly needed refactoring.

There was no time to refactor it during work hours due to other projects on my team’s plate, so I decided to refactor the bad code parts on my own time. Since I liked with the tooling environment I used for the project, I decided to use it for the refactor.

The Problem

Recreating this setup was a pain in the a**. Since I was only refactoring parts of the site code and not the entire site, the tooling need to be adjusted in some spots.

For example: I didn’t need all the dependencies listed in the package.json and bower.json files. I just needed the ones related to the code pieces I was refactoring, so I had to spend some time removing unneeded dependencies.

OK…that was more of an inconvenience than a pain in the a**. But setting up the CSS build process again? That sucked.

Again, I’m a fan of how well Bootstrap does responsive web design and based my LESS setup around that. My .less files were named based on Bootstraps’s pre-defined set of media queries, allowing me to add media query code based Bootstrap’s near-perfect setup.

So since Bootstrap has a media query called @media (min-width: 1200px), I have a file called min-width-1200.less that contains the following code:


@media (min-width: 1200px) {

}

And since it also has a media query called @media screen and (min-width: 768px), I have a file called screen-and-min-width-768.less that contains the following code:


@media screen and (min-width: 768px) {

}

You get the idea…

Like the package.json and bower.json files, the LESS files needed to be setup differently for the refactor. I only had to adjust two `.json files…I had to adjust 20 .less files.

Adjusting the .json dependencies was a manageable inconvenience…adjusting the .less files was a mind numbing sh*t-show.

I knew I would use this setup again and again, so containing it in a template I could use to start my projects seemed to make sense. I started building the template when I finished the refactor.

More Problems

The template started off as a bunch of files in a GitHub repo. It contained the .less and .json files and other things, like the Grunt & Gulp files and a .gitignore file.

Another set of problems soon appeared:

  • scaffolding this project from the repo wasn’t convenient. Starting a project with it meant I would need to git clone it to my machine. But that would download a folder with the files and I just needed the files. So if I was starting a project from scratch, I would have to rename the downloaded folder to match the project. Or, if I already started a project in another folder, I would have to copy the downloaded repo files to that other folder. All of this is doable…it’s just not convenient.
  • the template contained LESS files but I knew there would be instances where I would need to use another pre-processor. For example: Jekyll has built-in Sass integration so using Sass in that situation may make more sense. And what if I wanted to use another pre-processor like Stylus someday? Plus, I want to use Rework at some point because it lets you build a customizable pre-processor. So there may be times in the future when I don’t need LESS, Sass or another CSS pre-processor. Either way, the template needed to be flexible enough for me to add/remove pre-processors as I saw fit.
  • the template contained a .gitignore that listed some common files that should be ignored from Git commits. But my day job requires my working in a Microsoft .NET environment built around best practices codified over the course of a decade. One of these best practices is using Microsoft TFS for version control, not Git.
  • the template was configured really well for single page applications (SPAs), but I also do a fair amount of WordPress work. The template wasn’t really set up for that: some Gulp/Grunt plugins wouldn’t work, the CSS would need a slightly difference build-out process and the .gitignore would need some files added to it.
  • Making the tool work for .NET and WordPress environments meant Jade didn’t need to be add to a project by default.

So a downloadable template wouldn’t work because it would be too opinionated. It would assume that every project required the same tooling/dev environment setup…which isn’t true.

It was clear that I had to find a way to programmatically scaffold each project so I could configure it based on a set of passed (or not-passed) options. Yeoman is the current popular scaffolding app and I could have spent some time searching through Yeoman’s generators to find what I needed.

But what I wanted was too specific so I doubted the Yeoman could help. So I decided to use Node to try and build the tool for this.

The Tool-Building Process

I asked around and did some Google searches, trying to figure out the best way to do this. It took me two nights to get files and folders to either download or be created with a Node command.

From there, I became ridiculously anal-retentive and spent three weeks (THREE WEEKS) getting things to look and act how I wanted them to. This was mostly centered around logging: what messages were sent to the terminal console as the app progressed, what it took to get them to appear in the proper order, etc.

The Final Command

The end result of all this is kdz a Node tool that lets me scaffold a project and add small set of options. It downloads files from a GitHub repo, and it also creates files & folders….all in a programmatic way.

After you install it, typing kdz from anywhere in the terminal runs the --help command and outputs this:


  Usage: "kdz" [options] [command]


  Commands:

    app   scaffold a basic web application
    dt    delete "test-build" folder

  Options:

    -h, --help       output usage information
    -V, --version    output the version number
    -w, --wordpress  create a WordPress project
    -g, --gitignore  download ".gitignore" file
    -l, --less       download LESS files in "css-build"
    -s, --scss       download Sass files in "css-build"
    -t, --test       do a test scaffold in "test-build"

Just two commands for now…app and dt:

  • Running kdz app scaffolds out a SPA-like project by performing the following steps:
    • a build folder is created with css and js subdirectories.
    • a coffee folder is created and a main.coffee file is created inside of it.
    • a css-build folder is created with an imports folder, and an empty image-min folder is created (images that need to be minified go here).
    • bower.json, .bowerrc and STYLEGUIDE.md files are downloaded from the `source-shared-files` directory in the kdz repo.
    • SPA-like Gruntfile.js, gulpfile.js and package.json files are downloaded from the source-spa directory in the kdz repo.
  • The final build looks like this:
    
    ├── build
    |   ├── css
    |   └── js
            └── libs
    ├── coffee
    |   └── main.coffee
    ├── image-min
    ├── css-build
    |   └── imports
    ├── .bowerrc
    ├── bower.json
    ├── Gruntfile.js
    ├── gulpfile.js
    ├── package.json
    └── STYLEGUIDE.md
    
  • If the --test flag is passed to kdz app, a test-build folder is created, then a test scaffold goes into that folder. kdz dt is a quick way of deleting test-build after the testing is done and was used for kdz development purposes only. It plays no role in getting a web dev project done.
  • Along with --test and the standard --help and --version, there are a small set of options that can be passed to a command:
  • the --wordpress option scaffolds out a WordPress-like project. It performs almost the same tasks as kdz app with the following differences:
    • the build folder and its subdirectories are not created.
    • the Gruntfile.js, gulpfile.js and package.json files that are downloaded are more geared toward WordPress development and downloaded from the source-wordpress folder in the “kdz” repo.
    • a functions.php file is downloaded.
  • The final build looks like this:
    
    ├── coffee
    |   └── main.coffee
    ├── image-min
    ├── css-build
    |   └── imports
    ├── .bowerrc
    ├── bower.json
    ├── functions.php
    ├── Gruntfile.js
    ├── gulpfile.js
    ├── package.json
    └── STYLEGUIDE.md
    
  • the --gitignore option downloads a `.gitignore file from source-spa to the root folder by default. But if the --wordpress option is passed, `.gitignore will be WordPress-specific and downloaded from source-wordpress.
  • the --less option downloads LESS files from source-spa to css-build and css-build/imports by default. But if the --wordpress option is passed, the LESS files will be WordPress-specific and downloaded from source-wordpress. As mentioned, the .less files are named based on Bootstrap-defined media queries so the final build would look like this:
    
    css-build
     ├── style.less
     └── imports
         ├── all-transform-3d-webkit-transform-3d.less
         ├── bootstrap-override.less
         ├── for.less
         ├── globals.less
         ├── max-device-width-480-orientation-landscape.less
         ├── max-width-767.less
         ├── min-width-1200.less
         ├── min-width-768-max-width-991.less
         ├── min-width-768.less
         ├── min-width-992-max-width-1199.less
         ├── min-width-992.less
         ├── mixins.less
         ├── mobile_first.less
         ├── retina-media-queries.less
         ├── screen-and-max-width-767.less
         ├── screen-and-min-width-768.less
         ├── screen-webkit-min-device-pixel-ratio-0.less
         ├── screen-webkit-min-device-pixel-ratio-0
         └── variables.less
    
  • the --scss option does pretty much what --less does, except it downloads Sass files from source-spa to css-build and css-build/imports by default. But if the --wordpress option is passed, the Sass files will be WordPress-specific and downloaded from source-wordpress.

There are slight variations among the LESS and Sass builds based on my self-imposed rules. For example: the globals.less file exists for a reason and although there’s a for.less file, there’s no for.scss file.

Learn about these variations and rules over on the “Understand the Basic structure” section in repo’s style guide.

Behind The Scenes

Some interesting points about how “kdz” works behind the scenes:

  • Obviously, “kdz” was built with Node.
  • commander “Read about the npm commander module”) was the key package used to build this tool. It’s what I used to configure the commands and options. Thanks to [Ryan Ogden at Expand The Room](https://twitter.com/rogden83 “Visit Ryan Ogden on Twitter”) for turning me on to it.
  • q was used to manage JS Promises. Node runs things asynchronously and I had to make sure that a certain step didn’t run until some other steps ran before it. q properly managed that process.
  • The download and download-status modules work together to download files from the repo and display how the download is progressing.
  • chalk adds some pretty sweet coloring to the outputted console statements.

What’s Left To Do

A lot. The repo’s README has a TODO list of things I need/want to do, but here are the main things:

  • get “kdz” working on Windows: we’re at the point where there’s no good reason for a Node app to work on Unix-like systems only. Microsoft’s done far too much good work in getting Node to work on their operating systems and Azure: I think “kdz” should respect that.
  • make the Promises neater: I used Promises primarily to make sure the console logging happened in the proper order and while I’m (pretty) sure I implemented them properly, the code is spaghetti-like. This is the main reason why “kdz” is at v0.0.1 and I’d like to try and clean up the Promises before bumping a lot versions.
  • make a small library of methods: I spent a day looking at spots where the code was repetitive, then placing that code in a reusable function that all the spots could use. There are other spots that can use this: lots of spots use Node’s fs.open() method to check for the existence of a given file or folder. Placing that chunk in a library with a bunch of other reusable methods makes sense.
  • make the options run on their own: right now, the --gitignore, --less, --sass and --wordpress won’t run unless they’re passed as options to commands. In other words, running kdz app will scaffold a basic project but if I want to download Sass files sometime after that, I would have to run kdz app -s. Sass files should be able to be downloaded without the app command so I’m working on that.

Further Reading

A main reason that I didn’t write a tutorial was because there are so many good ones already out there. I found the Command-line utilities with Node.js article by Glynn Phillips to be the best one…I’d start there.

The Node API docs are also a read. It’s verbose in some spots but after pushing through it, I was able to write my own Node code where I would usually use a plugin…GOOD FOR ME!!!

Conclusion

“kdz” gives me the flexibility I needed. So back to the pre-processor conversation, I can add LESS and Sass files to a project by passing one of their respective options, but don’t have to pass any options if I want to use Rework.

And if every want to be able to configure Stylus or Jade for a project, I can just add an option to “kdz”. This will be really easy if I add a method library.

I’m glad creating “kdz” solved my problem but the BEST thing about building it was the amount of new Node knowledge I gained in the end. I had used Node before this and I understood it better once I started using Gulp.

But this project exposed me to ton of Node stuff. I gained a really good understanding of how Node interacts with a file system and all the quirks that come with exporting and requiring modules, along with some other things.

I also learned more about Promises and much they go hand-in-hand with Node. For all the issues with my Promise code, I better understand them now and see how they make Node development easier.

I still have more work to do: I really want to add more “Node stuff” to “kdz” instead of adding more “Node modules”. By that, I mean that I want to use things like .pipe() and process.nextTick() to see if they let me do things without requiring a bunch of pre-built npm modules.

But overall, creating “kdz” was one of the best web development experiences I’ve had in a while and can’t wait to continue developing it. If not to continue solving a problem, then to get Node to bend to my will some more.


Leave a Reply

Your email address will not be published. Required fields are marked *