OpenUserJS.org Establish a build/exec/deploy system

~~Fork a command line minifier for dev tools on the OpenUserJS account~~

~~This will be needed in a while to minify what we want to before deploying along with a deploy.sh that is kept with sizzle and maybe here... depends on the secrets that need to be kept.~~

Dev and GH source should remain unminified at all times... only time minified should be existing is on deployment/testing... this may require some extra time with making sure minification doesn't clobber anything... usually they are quite good at what they do by now but to be safe an extra testing measure should be done.

This is probably a long term issue since it's going to take some time to undo what's been done.

We could also use, instead, a shell script with git directly to a folder and then use some .sh scripts to move things to the appropriate locations... keeping in mind .gitignore . e.g. we really need a build.sh system and exec.sh type system.

This section may change constantly...

[x] VPS
[x] Initialization system... perpetual tweaking possibly. Invoked via $ npm install or $ node dev/init.js
- [ ] Look into https://github.com/alexanderGugel/ied as alternative
[x] Exec... Invoked via $ npm start or $ node app.js ... process management roughed in.
[x] Deploy... $ git pull on VPS
[ ] Minify CSS on the fly... partial... needs ~~enabling of cache system and placement on the VPS tree~~ with automated cleanup
[ ] Minify JavaScript on the fly... partial... ~~needs enabling of cache system and placement on the VPS tree~~ with automated cleanup... wants #432
[x] LESS caching (maybe... probable that express-minify is already doing this. Also bootstrap@4.x will use SASS)
[ ] Minify HTML
[ ] See if nock is viable for testing requests that aren't normally available in dev
[ ] ...

Jul 09 '14 05:07 Martii

This is where something like #97 Grunt or gulp.js would come in. As far as I'm concerned, all tools (including deploy scripts) should be in the repo in some sort of build or deploy folder to store temp files (auth info or other files). Either build tool can concatenate and minify JS, CSS, or precompile mustache templates.

Jul 09 '14 05:07 cletusc

Thanks Cletus... I'll start reading up on these, including grunts repo. We definitely seem to have calls over many CDNs. This really should stop and be centralized. A build system could manage some of this... hopefully without merge conflicts... but I see multiple .css/.js files from different CDNs with the same naming. While I'm sure those sites might be credible it's impending doom if they change something and kill OUJS for some reason.

Jul 09 '14 05:07 Martii

We can also use something along the lines of Bower in conjuction with these. You can set it up to pull a specific URL or repo, e.g. a known working version from a CDN or github, and then the build system can pull from that (just like anything in node_modules from npm install ...). When we need to update a version, it's as simple as changing the Bower info, then testing again. For this particular site, Grunt with Bower is probably the ideal combo.

Jul 09 '14 06:07 cletusc

CloudFlare seems to be a pretty popular CDN: https://www.cloudflare.com/features-cdn

They have this collection for rarer scripts: http://cdnjs.com/

At least if they go down, many other websites will be broken too. ;)

Obviously Google has their own CDN: https://developers.google.com/speed/libraries/devguide

Some advantages of using a CDN over self-hosting:

Faster load for new users, because they may already have the scripts (URLs) in their cache (from visits to previous websites using the CDN).
Faster load because the CDNs set huge expiry dates (so we don't have to fiddle with that).

Disadvantages:

If the CDN is unreachable by the user for whatever reason, it affects our site. If we self-host all the files, then we are self-contained.
Hosting the scripts ourselves might work better for tor users (not sure about that).

Jul 09 '14 14:07 joeytwiddle

I understand the only perk is for speed but with recent experiences these CDNs are more of a pain than a plus.

Disadvantages of a non-owned/managed CDN	Advantages of a non-owned/managed CDN
Lack of consistency in the code base. e.g. we are referencing CSS files that have the same name and are just slightly different but presumably the same purpose... this is bad.	Occasional and intermittent speed improvement
Lack of security esp. with .js hosted files
Lack of privacy. CDNs have always had this issue
On every machine in my nets the cache is cleared on browser close and open... occasionally during if it's initiated. Setting expiry dates isn't that difficult provided web standards are adhered to. ;)
Cloudflare is where we currently get our icons from and sometimes they never show up.
Ace editor has cross-domain issues when used on dev and occasionally on pro.
Their site goes down... we could be foo'd
Add-on Incompatibility of known blacklists

In general it's a bad practice if we don't own the management rights to the hosting space without some control of versioning. The weight of this merits hosting things ourselves or implementing project build controls. The files are relatively small so I don't think it's going to be much of an issue. The solution currently is to put these on our controlled site and have some sort of management to keep them up to date. I'd much rather we kill the site accidentally than someone else out there.

Jul 09 '14 19:07 Martii

@cletusc Hmm grunt requires a CLA to be signed if we ever need to contribute upstream. This is probably not a wise thing to mix with GPL... GPL will always take precedence in our project and usurping that clearly defined authority could be deemed as unsupported. There is also GHs licensing which this can conflict with. I'll still look into it some more but the gulp repo might be more viable ~~but don't know if it's Bower compatible. Do you know off hand?~~ Also if I was grunt I wouldn't be advertising that Adobe is one of their users... imeo they have some of most buggy closed source software around and we all get to pay for it usually. ;)

There's still always the option of doing this as natively as possible with git. Bash shell scripts can do just about anything and has a longer mtbf since it's been around since the dawn of terminal shells.

Related chatter:

#223

Jul 10 '14 18:07 Martii

@cletusc Cc: @sizzlemctwizzle Silly question... why would we want to use Bower which has a wrapped package for their system instead of $ npm directly which supports GH repo's? So far jQuery and ACE are on Bower but they are also on GH (jQuery on npm however ACE is not that I can find but I searched on GH for #281)... so couldn't we just use the npm package.json to do the downloading/updating/versioning? Does nodejitsu support GH downloads in package.json? We'll still need something to minify and I'm liking the docs for gulp since it doesn't use intermediate files as much. I assume that we can pull from the node_modules folder anything we want to a build or test folder, and deploy folder. (and/or reuse the dev folder)

$ npm install -h
npm install
npm install <pkg>
npm install <pkg>@<tag>
npm install <pkg>@<version>
npm install <pkg>@<version range>
npm install <folder>
npm install <tarball file>
npm install <tarball url>
npm install <git:// url>
npm install <github username>/<github project>

Jul 27 '14 05:07 Martii

so couldn't we just use the npm package.json to do the downloading/updating/versioning?

We could; the only downside is we wouldn't be able to do single files as with Bower.

I assume that we can pull from the node_modules folder

Or serve directly from the node_modules folder, if we ever wanted to go that route, but yes, npm/bower basically just downloads the package to node_modules/bower_components and you can deal with it just like any other file/folder.

Jul 27 '14 06:07 cletusc

the only downside is we wouldn't be able to do single files as with Bower.

It's unlikely that any dev would want skeleton code I think... deploying and testing could use gulp to scoot individual files around because they should be minified/uglified anyhow. I'd rather not bruise the original source when trying to track down a bug. Loading a minified file in takes FOREVER in any editor (save for VIM which is console based...even UltraEdit under Win has some issues sometimes) and I'm developing on a very new machine with a ton of processing power and memory... not to mention trying to search that. GH has some limitations too when the diff is too big but with smaller steps I think this can be managed. Thanks for your feedback.

Bower also creates a lot of weird hidden files... and somehow it created the same filename on my ext4 filesystem (not a symlink either)... so I'm a bit leery on that part. I also had to fill in the entire project information that is already in package.json and tell Bower that it was private (assume this needed to be done). So I'm leaning towards a -1 for Bower I think.

Jul 27 '14 06:07 Martii

I'd rather not bruise the original source when trying to track down a bug.

I didn't mean we pick and choose files that we need. I meant that some repos out there have their source in a single file, without a package.json; along the same lines as having a userscript in a single file, some projects do the same.

Loading a minified file in takes FOREVER in any editor

Sublime Text 3 on windows opens a 575562 character minified file in way under a second, and prettifies it in < 3 seconds to 731301 characters (I do this once every couple of days). Time to upgrade :P

Bower also creates a lot of weird hidden files

Not on my setup (Win7). I run npm install -g bower, then add a bower.json. Doing a bower install doesn't add any hidden files that I'm seeing.

I also had to fill in the entire project information that is already in package.json and tell Bower that it was private

If you wanted to add "private": true in there, you can, but we aren't publishing to the bower registry; FWIW we don't have private in our package.json, so would could technically publish the project to the npm registry. Our only commands with bower should be bower install though, so it shouldn't be an issue. This is the absolute bare minimum bower.json; just the name is needed and then your deps.

{
  "name": "openuserjs",
  "dependencies": {
    "jquery": "2.0"
  }
}

Whatever we do, we could use npm to manage the packages that will work with it (e.g. packages with a package.json), and Bower for cases where it doesn't.

Jul 27 '14 16:07 cletusc

Little short there for Bower config... bare bones is:

{
  "name": "OpenUserJS.org",
  "version": "0.1.3",
  "homepage": "https://github.com/OpenUserJs/OpenUserJS.org",
  "description": "An open source user scripts repo built using Node.js",
  "main": "./app.js",
  "moduleType": [
    "node"
  ],
  "keywords": [
    "user",
    "script",
    "userscript"
  ],
  "authors": [
    "sizzlemctwizzle"
  ],
  "license": "GPL-3.0+",
  "private": true,
  "ignore": [
    "**/.*",
    "node_modules",
    "bower_components",
    "test",
    "tests"
  ],
  "dependencies": {
    "ace-builds": "~1.1.5",
    "jquery": "~2.1.1"
  }
}

Having to maintain two separate configs is going in the opposite direction of what this issue is intended. I'm also thinking none of them are going to address rereferencing things that get suffixed e.g. filename.min.js vs filename.js in our actual source.

Let's not get into a debate on editors. Besides I have no intention of reading minified code for any reason. ;) :)

I meant that some repos out there have their source in a single file

The only time this currently applies that I've seen is with some CSS that we pull, for no apparent reason. I would think we would want to handle this ourselves. The google fonts were a prime example of what not to do. Fonts almost never change. I'm using some ttf's from the early 90's and they still work like a charm on every system... granted wrappers have changed a bit for open fonts but they still don't change.

Jul 27 '14 17:07 Martii

Little short there for Bower config

Not if we are just consuming deps. Absolute bare minimum is what I posted--see the bower.json spec; all the other stuff is if you are publishing to the Bower registry, of which we are not.

Besides I have no intention of reading minified code for any reason.

Unfortunately, I have to for a userscript; gotta see how the site does certain things to prevent conflicts :D

Jul 27 '14 18:07 cletusc

without a package.json;

There's also this. So the rule would be download whatever is there instead of a versioned item. This seems like a recipe for disaster with multiple file management.

I'm still curious why we have CSS files that are nearly the same but slightly different referenced from cloudflare... I would think that our CSS would be the same and we would want to track changes here on GH instead of blindly following some CDN out there that only has one file on it that may or may not be available at the time of querying.

gotta see how the site does certain things to prevent conflicts

Do you mean if OUJS clobbers something with minification? Those projects that minify are using the same things we could be using. The minified code shouldn't be obfuscated or different. It was not easy determining the Ace version we have locally since those comment headers were stripped out.

Absolute bare minimum is what I posted

Well Bower install started what looked like a TCL/TCK script and starting asking me questions. New devs to the project would need to be instructed on how to use Bower and what not to pick. I think it could be an option chosen by a dev... we could locally ignore those folders but I don't think we should rely on CDNs unless we have to. In most cases it's best to track the changes here on GH and figure things out for ourselves. We can of course preview what others are doing that's why it's called a SCM repo here. :) We're still going to need some accelerator shell scripts too.

Jul 27 '14 18:07 Martii

Do you mean if OUJS clobbers something with minification?

No, I mean I have to dig through this and this on a regular basis for this. Was off-topic.

Jul 27 '14 18:07 cletusc

UPDATE

As I've been commenting and referencing/alluding galore for this issue this is a huge undertaking as well as a work in progress one step at a time.

With the recent discovery that $ npm currently has issues with GH urls as dependencies I'm tacking on tracking upstream because that portion needs to be fixed in npm

Target test: ~~$ npm install git://github.com/github/octicons.git~~ That stream added package.json in a PR.

Also bumping the source tree down one level to src is probably what needs to be done in which @sizzlemctwizzle will need to be directly involved with this since deployment is affected as well.

This is where I am at.

Some rambling notes that are probably going to be changing in the comment to reduce the noise level a bit:

init or similar like build
exec
test
deploy

Oct 23 '14 16:10 Martii

@cletusc Cc: @sizzlemctwizzle and anyone else too

Does anyone have a preference on the file cache for the release version of express-minify? This affects dev as well as pro.

Possible locations are:

./.cache/express-minify... (static or dynamically created?)
SystemTempSpecialFolder/openuserjs.org/cache/express-minify... (dynamically created... optional oujs short name instead of full?)
HomeDirectorySpecialFolder/.cache/openuserjs.org/express-minify... (dynamically created... optional oujs short name instead of full?)
Other physical disk location... (please specify)

If no response something will be chosen... of course it can be changed at a later date.

Mar 01 '16 00:03 Martii