JavaScript Jabber JSJ 266 NPM 5.0 with Rebecca Turner
Jun 20, 2017
41:47
On today’s episode of JavaScript Jabber, Charles Max Wood and panelist Joe Eames chat with Rebecca Turner, tech lead for https://www.npmjs.com/, a popular Javascript package manager with the worlds largest software registry. Learn about the newly released NPM 5 including a few of the updated features. Stay tuned![1:58] Was the release of node JS 8 tied to NPM5?
- Features in NPM5 have been in planning for 2 years now.
- Planned on getting it out earlier this year.
- Node 8 was coming out and got pushed out a month.
- Putting NPM5 into Node 8 became doable.
- Pushed really hard to get NPM5 into https://nodejs.org/en/blog/release/v8.0.0/ so that users would get NPM5 and updates to NPM5.
[2:58] Why would it matter? NPM doesn’t care right?
- Right you can use NPM5 with any version of node.
- Most people don’t update NPM, but upgrade Node.
- So releasing them together allowed for when people updated Node they would get NPM 5.
[3:29] How does the upgrade process work if you’re using NVM or some node version manager?
- Depends. Different approaches for each
- NVM gets a fresh copy of Node with new globals. NVM5 and Node 8 are bundled.
- For some, If you manually upgrade NVM you’ll always have to manually. It will keep the one you manually upgraded to.
[4:16] Why NPM 5?
- It’s night and day faster.
- 3 to 5 times speed up is not uncommon.
- Most package managers are slow.
- NPM 5 is still growing. Will get even faster.
[5:18] How did you make it faster?
- The NPM’s cache is old. It’s very slow. Appalling slow.
- Rewrote cache
- Saw huge performance gains
[5:49] What is the function of the cache?
- Cache makes it so you don’t have to reinstall modules from the internet.
- It has registry information too.
- It will now obey http headers for timing out cache.
[6:50] Other things that made it faster?
- Had a log file for a long time. It was called https://docs.npmjs.com/cli/shrinkwrap.
- NPM 5 makes it default.
- Renamed it to packagelog.json
- Exactly like shrinkwrap package file seen before
- In combo with cache, it makes it really fast.
- Stores information about what the tree should look like and it’s general structure.
- It doesn’t have to go back and learn versions of packages.
[7:50] Can you turn the default Packagelog.json off?
- Yes. Just:
- Set packagelog=false in the npmrc
[8:01] Why make it default? Why wasn’t it default before?
- It Didn’t have it before. Shrinkwrap was added as a separate project enfolded in NPM and wasn’t core to the design of NPM.
- Most people would now benefit from it. Not many scenarios where you wouldn’t want one.
- Teams not using the same tools causes headaches and issues.
[9:38] Where does not having a lock show up as a problem?
- It records the versions of the packages installed and where NPM put them so that when you clone a project down you will have exactly the same versions across machines.
- Collaborators have the exact same version.
- Protects from issues after people introduce changes and patch releases.
- NPM being faster is just a bonus.
- Store the sha512 of the package that was installed in the glock file so that we can verify it when you install. It’s Bit for bit what you had previously.
[11:12] Could you solve that by setting the package version as the same version as the .Json file?
- No. That will lock down the versions of the modules that you install personally, not the dependancies, or transitive dependancies.
- Package log allows you to look into the head of the installer. This is what the install looks like.
[12:16] Defaulting the log file speed things up? How?
- It doesn’t have to figure out dependences or the tree which makes it faster.
- Shrinkwrap command is still there, it renames it to shrinkwrap but shrinkwrap cannot be published.
- For application level things or big libraries, using shrinkwrap to lock down versions is popular.
[13:42] You’ve Adopted specifications in a ROC process. When did you guys do that?
- Did it in January
- Have been using them internally for years. Inviting people into the process.
- Specifications
- Written in the form of “Here is the problem and here are the solutions.”
- Spec folder in NPM docs, things being added to that as they specify how things work.
- Spec tests have been great.
[14:59] The update adds new tools. Will there be new things in registry as well?
- Yes.
- Information about a package from registry, it returns document that has info about every version and package json data and full readme for every version.
- It gets very large.
- New API to request smaller version of that document.
- Reduces bandwidth, lower download size, makes it substantially faster.
- Used to be hashed with sha1, With this update it will be hashed with sha512 as well as sha1 for older clients.
[16:20] Will you be stopping support for older versions?
- LTS version of NPM was a thing for a while. They stopped doing that.
- Two models, people either use whatever version came with Node or they update to the latest.
- The NPM team is really small. Hard to maintain old NPM branches.
- Supports current versions and that’s pretty much it.
- If there are big problems they will fix old versions. Patches , etc.
[17:36] Will there ever be problems with that?
- Older versions should continue to work. Shouldn’t break any of that.
- Can’t upgrade from 0.8.
- It does break with different Node version
- Does not support Node versions 0.10 or 0.12.
[18:47] How do you upgrade to NPM?
- sudo npm install -gmpm
- Yes, you may not need sudo. depend on what you’re on.
[19:07] How long has it been since version 4?
- Last October is when it came out.
[19:24] Do you already have plans for version 6?
- Yes!
- More releases than before coming up.
- Finally deprecating old features that are only used in a few packages out of the whole registry.
- Running tests on getting rid of things.
[20:50] Self healing cache. What is it and why do we want it?
- Users are sometimes showing up where installs are broken and tarbols are corrupted.
- This happens sometimes with complicated containerization setups makes it more likely. It’s unclear where the problem actually is.
- https://www.npmjs.com/package/cacache - content addressable cache. Take the hash of your package and use it to look up address to look it up in the cache.
- Compares the Tarbol using an address to look it up in the cache.
- Compares to see if it’s old. Trashes old and downloads updated one.
- Came out with the cache. Free side effect of the new cache.
[23:14] New information output as part of the update?
- NPM has always gave back you the tree from what you just installed.
- Now, trees can be larger and displaying that much information is not useful.
- User patch - gives you specifically what you asked for.
- Information it shows will be something like: “I installed 50 items, updated 7, deleted 2.”
[24:23] Did you personally put that together?
- Yes, threw it together and then got feedback from users and went with it.
- Often unplanned features will get made and will be thrown out to get feedback.
- Another new things ls output now shows you modules that were deduped. Shows logical tree and it’s relationships and what was deduped.
[25:27] You came up to node 4 syntax. Why not go to node 8?
- To allow people with just node 4 be able to use NPM.
- Many projects still run Node 4. Once a project has been deployed, people generally don’t touch it.
[26:20] Other new features? What about the File Specifier?
- File specifier is new. File paths can be in package json, usually put inside pointing to something inside your package.
- It will copy from there to your node modules.
- Just a node module symlink.
- Much faster. Verifiable that what’s in your node modules matches the source. If it’s pointing at the right place it’s correct. If not, then it’s not.
- Earlier, sometimes it was hard to tell.
[27:38] Anything else as part of the NPM 5 release? Who do you think will be most affected by it?
- For the most part, people notice three things:
- 1st. no giant tree at the end
- 2nd. Much faster
- 3rd. Package lock.
[28:14] If it’s locked, how do you update it?
- Run npm installer and then npm update
- Used to be scary, but works well now.
- Updates to latest semver, matches semver to package json to all node modules.
- Updates package lock at the same time
- Summary in Git shows what’s changed.
[28:59] Did Yarn come into play with your decisions with this release?
- The plans have been in play for a long time for this update.
- https://yarnpkg.com/en/ inclusion of similar features and the feedback was an indicator that some of the features were valuable.
[29:53] Other plans to incorporate features similar to yarn?
- Features are already pretty close.
- There are other alternative package managers out there.
- PMPM interesting because when it installs it doesn’t copy all the files. It c
Become a supporter of this podcast: https://www.spreaker.com/podcast/javascript-jabber--6102064/support.
- Features in NPM5 have been in planning for 2 years now.
- Planned on getting it out earlier this year.
- Node 8 was coming out and got pushed out a month.
- Putting NPM5 into Node 8 became doable.
- Pushed really hard to get NPM5 into https://nodejs.org/en/blog/release/v8.0.0/ so that users would get NPM5 and updates to NPM5.
[2:58] Why would it matter? NPM doesn’t care right?
- Right you can use NPM5 with any version of node.
- Most people don’t update NPM, but upgrade Node.
- So releasing them together allowed for when people updated Node they would get NPM 5.
[3:29] How does the upgrade process work if you’re using NVM or some node version manager?
- Depends. Different approaches for each
- NVM gets a fresh copy of Node with new globals. NVM5 and Node 8 are bundled.
- For some, If you manually upgrade NVM you’ll always have to manually. It will keep the one you manually upgraded to.
[4:16] Why NPM 5?
- It’s night and day faster.
- 3 to 5 times speed up is not uncommon.
- Most package managers are slow.
- NPM 5 is still growing. Will get even faster.
[5:18] How did you make it faster?
- The NPM’s cache is old. It’s very slow. Appalling slow.
- Rewrote cache
- Saw huge performance gains
[5:49] What is the function of the cache?
- Cache makes it so you don’t have to reinstall modules from the internet.
- It has registry information too.
- It will now obey http headers for timing out cache.
[6:50] Other things that made it faster?
- Had a log file for a long time. It was called https://docs.npmjs.com/cli/shrinkwrap.
- NPM 5 makes it default.
- Renamed it to packagelog.json
- Exactly like shrinkwrap package file seen before
- In combo with cache, it makes it really fast.
- Stores information about what the tree should look like and it’s general structure.
- It doesn’t have to go back and learn versions of packages.
[7:50] Can you turn the default Packagelog.json off?
- Yes. Just:
- Set packagelog=false in the npmrc
[8:01] Why make it default? Why wasn’t it default before?
- It Didn’t have it before. Shrinkwrap was added as a separate project enfolded in NPM and wasn’t core to the design of NPM.
- Most people would now benefit from it. Not many scenarios where you wouldn’t want one.
- Teams not using the same tools causes headaches and issues.
[9:38] Where does not having a lock show up as a problem?
- It records the versions of the packages installed and where NPM put them so that when you clone a project down you will have exactly the same versions across machines.
- Collaborators have the exact same version.
- Protects from issues after people introduce changes and patch releases.
- NPM being faster is just a bonus.
- Store the sha512 of the package that was installed in the glock file so that we can verify it when you install. It’s Bit for bit what you had previously.
[11:12] Could you solve that by setting the package version as the same version as the .Json file?
- No. That will lock down the versions of the modules that you install personally, not the dependancies, or transitive dependancies.
- Package log allows you to look into the head of the installer. This is what the install looks like.
[12:16] Defaulting the log file speed things up? How?
- It doesn’t have to figure out dependences or the tree which makes it faster.
- Shrinkwrap command is still there, it renames it to shrinkwrap but shrinkwrap cannot be published.
- For application level things or big libraries, using shrinkwrap to lock down versions is popular.
[13:42] You’ve Adopted specifications in a ROC process. When did you guys do that?
- Did it in January
- Have been using them internally for years. Inviting people into the process.
- Specifications
- Written in the form of “Here is the problem and here are the solutions.”
- Spec folder in NPM docs, things being added to that as they specify how things work.
- Spec tests have been great.
[14:59] The update adds new tools. Will there be new things in registry as well?
- Yes.
- Information about a package from registry, it returns document that has info about every version and package json data and full readme for every version.
- It gets very large.
- New API to request smaller version of that document.
- Reduces bandwidth, lower download size, makes it substantially faster.
- Used to be hashed with sha1, With this update it will be hashed with sha512 as well as sha1 for older clients.
[16:20] Will you be stopping support for older versions?
- LTS version of NPM was a thing for a while. They stopped doing that.
- Two models, people either use whatever version came with Node or they update to the latest.
- The NPM team is really small. Hard to maintain old NPM branches.
- Supports current versions and that’s pretty much it.
- If there are big problems they will fix old versions. Patches , etc.
[17:36] Will there ever be problems with that?
- Older versions should continue to work. Shouldn’t break any of that.
- Can’t upgrade from 0.8.
- It does break with different Node version
- Does not support Node versions 0.10 or 0.12.
[18:47] How do you upgrade to NPM?
- sudo npm install -gmpm
- Yes, you may not need sudo. depend on what you’re on.
[19:07] How long has it been since version 4?
- Last October is when it came out.
[19:24] Do you already have plans for version 6?
- Yes!
- More releases than before coming up.
- Finally deprecating old features that are only used in a few packages out of the whole registry.
- Running tests on getting rid of things.
[20:50] Self healing cache. What is it and why do we want it?
- Users are sometimes showing up where installs are broken and tarbols are corrupted.
- This happens sometimes with complicated containerization setups makes it more likely. It’s unclear where the problem actually is.
- https://www.npmjs.com/package/cacache - content addressable cache. Take the hash of your package and use it to look up address to look it up in the cache.
- Compares the Tarbol using an address to look it up in the cache.
- Compares to see if it’s old. Trashes old and downloads updated one.
- Came out with the cache. Free side effect of the new cache.
[23:14] New information output as part of the update?
- NPM has always gave back you the tree from what you just installed.
- Now, trees can be larger and displaying that much information is not useful.
- User patch - gives you specifically what you asked for.
- Information it shows will be something like: “I installed 50 items, updated 7, deleted 2.”
[24:23] Did you personally put that together?
- Yes, threw it together and then got feedback from users and went with it.
- Often unplanned features will get made and will be thrown out to get feedback.
- Another new things ls output now shows you modules that were deduped. Shows logical tree and it’s relationships and what was deduped.
[25:27] You came up to node 4 syntax. Why not go to node 8?
- To allow people with just node 4 be able to use NPM.
- Many projects still run Node 4. Once a project has been deployed, people generally don’t touch it.
[26:20] Other new features? What about the File Specifier?
- File specifier is new. File paths can be in package json, usually put inside pointing to something inside your package.
- It will copy from there to your node modules.
- Just a node module symlink.
- Much faster. Verifiable that what’s in your node modules matches the source. If it’s pointing at the right place it’s correct. If not, then it’s not.
- Earlier, sometimes it was hard to tell.
[27:38] Anything else as part of the NPM 5 release? Who do you think will be most affected by it?
- For the most part, people notice three things:
- 1st. no giant tree at the end
- 2nd. Much faster
- 3rd. Package lock.
[28:14] If it’s locked, how do you update it?
- Run npm installer and then npm update
- Used to be scary, but works well now.
- Updates to latest semver, matches semver to package json to all node modules.
- Updates package lock at the same time
- Summary in Git shows what’s changed.
[28:59] Did Yarn come into play with your decisions with this release?
- The plans have been in play for a long time for this update.
- https://yarnpkg.com/en/ inclusion of similar features and the feedback was an indicator that some of the features were valuable.
[29:53] Other plans to incorporate features similar to yarn?
- Features are already pretty close.
- There are other alternative package managers out there.
- PMPM interesting because when it installs it doesn’t copy all the files. It c
Become a supporter of this podcast: https://www.spreaker.com/podcast/javascript-jabber--6102064/support.