files: list all required files for build tools

[desandro]

Since resolving main in #43, we've thrown a wrench in the gears for build tools, most prevelantly Wiredep. Wiredep needs a list of all the files to be included when you install the package. Previously, main was being widely used for this purpose. Now main should no longer lists all the files (The entry-point files necessary to use your package. One per filetype). We need another field.

These two proposals are my first shot at it. Your thoughts appreciated :watermelon:

Proposal 1: files flat list

files is an array of all the files required to use the package. Filepaths may include globs.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

We recommend listing source files in files, rather than compiled distribution files.

"files": [
  "js/motion.js",
  "js/run.js",
  "js/walk.js",
  "sass/motion.scss",
  "sass/run.scss",
  "sass/walk.scss",
  "img/motion.png",
  "img/walk.png",
  "img/run.png",
  "fonts/icons.woff2",
  "fonts/icons.woff"
]

Most likely you would list globs for multiple files of the same type.

"files": [
  "js/*.js",
  "sass/*.scss",
  "img/*.*",
  "fonts/*.*"
]

Pros: Simple specification, easier for developers to adopt. Don't have to worry on Bower spec side about asset-type specification. Cons: Aside from file extension, it's not clear what files do what. For example, a package make require icon.svg — is this file a font or should be it used as an image?

We leave it up to build tools on how they make use of the array. Some might only be concerned about .js and .css files. For globs, they'd have to expand the glob and do the logic on their side.

Proposal 2: asset-type object

Taking from #21

files is an object listing all the files required to use the package. files are listed as arrays for each type of asset.

• scripts: Script files like JavaScript .js or CoffeeScript .coffee
• styles: Style files like CSS .css, Sass .scss, or LESS .less
• images: ...
• fonts: ...
• data
• templates
• (Others — we'd have to determine this spec)

Filepaths may include globs. yada yada yada ....

"files": {
  "scripts": [
    "js/motion.js",
    "js/run.js",
    "js/walk.js",
  ],
  "styles": [
    "sass/motion.scss",
    "sass/run.scss",
    "sass/walk.scss",
  ],
  "images": [
    "img/motion.png",
    "img/walk.png",
    "img/run.png",
  ],
  "fonts": [
    "fonts/icons.woff2",
    "fonts/icons.woff"
  ]
}

Pro: Clear about what files do what. Cons: More complex. We'd have to figure out the list of types. Might be confusing files that do not have an explicit type, like if there's a video file and we don't have a videos field.

Fun fact: recommended way back in 2013 https://github.com/bower/bower/issues/935#issuecomment-26923813

---

files would live along side main, not overwrite it. I'm not sure if it should be required or part of the bower init flow at this point.

Currently, I prefer Proposal 1. Easier for authors and for Bower spec.

Related issues

• taptapship/wiredep#180
• twbs/bootstrap#16663
• metafizzy/outlayer#33

[HeroBart]

So the new "files" will hold all source files, and "main" will hold compiled versions again? If so, I'd vote for proposal 1, it's simple enough.

[kkirsche]

With proposal 2 I think you have the potential to accept valid MIME types as the category list. By doing this, it's not up to you to figure out what kinds of files people need and you offload that to a different organization where it is getting close monitoring. Personally, I prefer proposal 2 as I feel there is more clarity which would be useful to users who are new or inexperienced with a project.

[alber70g]

Does this mean that the dist/movement.css used in the example of the current spec can be in main? Because if that's the case, nothing really changes. Except the hard-exclude of movement.css is out of the way.

[kkirsche]

@Alber70g It doesn't read that way to me — this would put them in a separate files block. main would retain source files only is my understanding of the OP's post.

[HeroBart]

So we can expect half a year of wiredep not working and dependency mayhem... Nice.

[alber70g]

files is an array of all the files required to use the package. Filepaths may include globs.

files differs from main in that main only lists single files per filetype, whereas files should include every file.

We recommend listing source files in files, rather than compiled distribution files.

The second line implies that we should keep only ONE file per filetype in main The last line implies that we should put into files what is now in main in a sense that .less (or scss) is not allowed to be in main (which is the case, by looking at the current spec).

[nmalaguti]

Who is files meant to be consumed by? Any sufficiently advanced packer should be able to parse the assets listed in main and resolve all other dependencies (scripts, styles, images, fonts, etc.).

The problem is that naive dependency resolvers like wiredep don't have much file specific knowledge, they just want to resolve paths and group assets by extension. Depending on the file extension, they are injected into html or returned in an object (organized by extension/dependency) for further tools to consume.

There are several kinds of information I would find useful from a package (property names aren't important here):

• main: entry points for various file types that can be directly referenced from a browser. Not all file types support imports (e.g. ES5) and therefore concat + minification may be desirable for some.
• metadata: e.g. .d.ts or .less files with typing or variable information, but that compile down to nothing. Useful when doing development.
• sources: a list of source entry point files in the component that are needed for compilation (if no sources use imports/exports, then all source files should be listed). Don't list intermediate representations (e.g. ts -> js -> min.js, the js files should not be listed).
• assets: a list of static assets that are referenced by various other files (e.g. fonts, images, etc.) that should be copied into production so that unneeded files (sources, metadata) can be filtered out.

It may also be useful to offer different collections of files in the package that can be combined together with main as needed by the consumer (think various sizes, legacy support, languages):

{
  "main": [
    "js/motion-base.js"
  ],
  "options": {
    "lite": [
      "js/motion-lite.js"
    ],
    "full": [
      "js/motion-full.js"
    ],
    "huge": [
      "js/motion-huge.js"
    ],
    "legacy": [
      "js/motion-legacy.js"
    ]
  }
}

There would likely need to be some way for the consumer to specify which option(s) they want to be combined with main.

I think this gives enough information for a consumer to take the bower.json and pipe the various file lists into their tools of choice.

• metadata is used by the consumer to build their own projects/do development
• sources can go to a compiler (and then main should be ignored as you don't want to include stuff twice)
• main + options are injected as paths into html (or whatever you want to do with file paths)
• assets are retained when shipping to production so all unneccessary files can be stripped out (I already minified all js and therefore don't need to ship it)

[cvrebert]

• Re: Proposal 1: files flat list:
  • Couldn't tools just do ls -R . and then subtract out the ignore entries and get the same result? If yes, I don't see the point; Bower or the tools could just generate the list programmatically at runtime instead.
• Re: Proposal 2: asset-type object:
  • Better, but I think it would be good to tie precompiled CSS to the preprocessor file that it originated from, which this proposal doesn't do.
  • Example/strawman:

"files": {
  "styles": {
    "dist/css/bootstrap.css": "less/bootstrap.less",
    "dist/css/bootstrap-theme.css": "less/theme.less"
  }
}

Other issues:

• What about bootstrap.min.css.map?
  • Can we assume the <foo>.map naming convention and not explicitly list these and just have tools check for them?
• What about bootstrap.min.css and bootstrap.min.js?
  • Can we similarly assume the <foo>.min.<bar> naming convention?

[ezraroi]

I think proposal 1 is better, build tools will easily adopt it I believe

[stephenplusplus]

Proposal one works for wiredep :+1: I wouldn't vote for globs, however. It would just slow things down and the publisher should be able to explicitly list what files are important per release. They've been doing it with main already anyway.

[sheerun]

+1 for Option 1 files, that is compatible with both wiredep and NPM specification...

Mime-types of Option 2 can be easily extracted automatically..

I'd go without globs, but with special behavior of npm:

The "files" field is an array of files to include in your project. If you name a folder in the array, then it will also include the files inside that folder. (Unless they would be ignored by another rule.)

[cvrebert]

@sheerun What becomes the purpose/use of ignore then?

[sheerun]

@cvrebert It would be the same as .npmignore, but without separate file:

You can also provide a ".npmignore" file in the root of your package, which will keep files from being included, even if they would be picked up by the files array. The ".npmignore" file works just like a ".gitignore".

[cvrebert]

@sheerun Alrighty.

@paulmillr Does Brunch have any thoughts about the proposals?

[donaldpipowitch]

I'm fine with globs.

However... we should note what problems we want to solve here... and which not.

1. What is with minified/optimized code/assets?
2. What is with "dynamic" code/assets as seen in https://github.com/angular/bower-angular-i18n where we only need one locale-specific file?
3. What is our solution to different builds (compatible-modes for older browsers, etc.)? A current problem for RxJS https://github.com/Reactive-Extensions/RxJS/issues/724.
4. What is with source maps?

Do we answer these questions here or in a different place? I would like to solve real problems here.

[HeroBart]

By the way, while we're at it, why don't we choose a more descriptive name, such as "sourceFiles" or "source".

[sheerun]

I've just read why you want files introduced..

I think Bower made mistake by encouraging to include only source files in main. Maybe we should roll-back a little and accept this pull-request: https://github.com/bower/bower.json-spec/pull/46/files

That said, I think files is good idea anyway, as a more explicit way of listing all files to include in package (not main entry-points, that's what main is for). It can be useful for packages without main files, like CSS libraries.

[HeroBart]

+1 for @sheerun, please do!

[donaldpipowitch]

Would this exclude Less files from "main"? I don't know if I like that :/

[sheerun]

@donaldpipowitch No, less can be library entry-point as well. If you use less, you probably would be more interested in consuming .less main files. If you use sass, you would be interested in consuming .sass or .scss. If you use plain html, you want to use .css main files. Bower for better of worse tries to be language-agnostic..

[donaldpipowitch]

Okay, thanks. I asked because of this line in the pull request:

"This means that the files can be used without compiling for use by the consuming party."

[sheerun]

Technically .less and .scss files don't need compilation to be consumed by less, sass tools... But maybe it needs clarification.

[alber70g]

@sheerun @donaldpipowitch I've updated my pull request to give more clarification about 'high level' sources (less, sass, ts etc): https://github.com/Alber70g/bower.json-spec/commit/a93cd95e639d6bb421ba4ac5f75e791256e650bb

[sheerun]

@Alber70g Now it's better, but you probably linked wrong commit. This is correct:

https://github.com/bower/bower.json-spec/pull/46/files

Note that images / fonts are not entry-points because you can't just "use them by default". They always need to be included from another resource (HTML / CSS). That's why they should be listed only in files (my opinion).

[donaldpipowitch]

Thank you all. That's better!

[alber70g]

@sheerun Yea, I linked the commit instead of the pull request

Well, you can reference fonts as library. For example if you are building a new 'bootstrap' and you want to include 3rd party fonts, they can be part of wiredep when you inject them into your less

For example:

newBootstrap.less

/*inject:fonts*/
/*endinject*/

[paulmillr]

@cvrebert I don't like idea of including minified files in.

Brunch minifies the result. Non-minified ver is useful for debugging.

[desandro]

Thanks for all the input.

I'm not keen on rolling-back main. I see a functional difference between main and files:

• main is for entry-point files. Useful for RequireJS, LESS, Sass, and tools that read file source to pull in additional dependencies.
• files is for build tools that do not read file sources, like Wiredep, and need to know all the files that should be included.

---

Build tools should build from source files.

I see this as the root dilemma, as exhibited by the Bootstrap/Wiredep issue. Bootstrap's source styles are in .less files. Build tools should build from those .less files. Packages should avoid pointing to compiled files — i.e. Bootstrap's dist .css file. Wiredep in fact does work with .less files. A LESS compiler can be added to the build workflow, either by Wiredep, or on the users build process' side.

[nmalaguti]

@desandro

.less files are a relatively simple transformation and a quick compilation step. What about languages like TypeScript that are more expensive to compile? If a component author wants to write their component in TypeScript, should all downstream users be required to setup the TypeScript compiler in order to use the component, even though they only care about the javascript?

[donaldpipowitch]

I don't know if it is practical, but maybe we need two bower modules then. One exposing the TypeScript and one exposing the JavaScript...? Or it should just add the .d.ts and .js, but never .ts.