[build-script] Reduce the size of development snapshots #801
Conversation
| @@ -2249,10 +2249,10 @@ if [[ "${INSTALLABLE_PACKAGE}" ]] ; then | |||
| fi | |||
|
|
|||
| (cd "${INSTALL_DESTDIR}" && | |||
| tar -c -z -f "${INSTALLABLE_PACKAGE}" "${TOOLCHAIN_PREFIX/#\/}") | |||
| tar -c -J -f "${INSTALLABLE_PACKAGE}" "${TOOLCHAIN_PREFIX/#\/}") | |||
There was a problem hiding this comment.
This is a branch specific to OS X. tar on OS X does not support the -J flag.
There was a problem hiding this comment.
My version of OS X does. I don't think my version of tar is from Homebrew; but perhaps it is? I'm fine with removing it if that is your desire. Just thought I'd point it out.
I did at least test the command on my install OS X El Capitan 10.11.2 (15C50) and it supported the -J flag.
$ tar --help
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
-c Create -r Add/Replace -t List -u Update -x Extract
Common Options:
-b # Use # 512-byte records per I/O block
-f <filename> Location of archive
-v Verbose
-w Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
<file>, <dir> add these items to archive
-z, -j, -J, --lzma Compress archive with gzip/bzip2/xz/lzma
--format {ustar|pax|cpio|shar} Select archive format
--exclude <pattern> Skip files that match pattern
-C <dir> Change to <dir> before processing remaining files
@<archive> Add entries from <archive> to output
List: tar -t [options] [<patterns>]
<patterns> If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
<patterns> If specified, extract only entries that match
-k Keep (don't overwrite) existing files
-m Don't restore modification times
-O Write entries to stdout, don't restore to disk
-p Restore permissions (including ACLs, owner, file flags)
bsdtar 2.8.3 - libarchive 2.8.3
There was a problem hiding this comment.
Indeed, it does. My bad -- I was checking the man page.
|
Is it supported on RHEL 7 / CentOS 7? |
|
I don't have objections to this change, but we need to coordinate with @shahmishal to update other automation scripts that handle these packages. |
|
(I'm trying to get a CentOS 7 VM image now to test on it) Frankly I don't have a strong opinion either way. If it breaks too many processes or things that I cannot see from the outside then it won't break my heart to see this closed. I use it on my own local daily builds just to keep the file storage down. The patch is small enough for me to maintain myself without it being subsumed into the mainline. I thought the community may benefit from the patch so I submitted it. |
|
If the LZMA2 compression is supported everywhere, then using it does not "break" processes, it just requires changes to them in places that somehow assume |
|
I have now built a CentOS 7.1 VM and can confirm that it also supports LZMA2 compression. |
|
@RLovelett Sounds great, thank you for checking. Let's coordinate landing this change with @shahmishal after the break. |
Currently development snapshots for Linux are provided as `*.tar.gz` files (Gzip compression). This patch switches the archive format to `*.tar.xz` (LZMA2 compression). There are lots of discussions out there that talk about why LZMA2 format might be preferred over Gzip so I'll omit them here. For this project, the major benefit is an [approximate 46% reduction](http://www.wolframalpha.com/input/?i=percent+difference&f1=60306748&f=PercentDifference.number1_60306748&f2=88476335&f=PercentDifference.number2_88476335) in the source snapshots provided on Swift.org (from ~85MB to ~58MB). The trade-off is that it does add a little bit of time to the build, on my machine it amounts to another minute or so. Considering that the over-all build currently takes longer than an hour I would consider this addition "in-the-noise" and a fair trade off. Your experience and opinion may differ. Additionally another often cited negative about LZMA2 vs Gzip is compatibility. Both of the targeted Linux builds `tar` utilities support LZMA2. My expectation is that the number of downloads is much much greater than than the number of people building from source and packaging. With that in mind this seems like an easy gain with little downside.
|
@shahmishal , do you have concerns or should we go ahead with this? |
|
Before we merge this, will need to make sure our backend (code sign system and swift.org) is ready to handle it. |
|
@swift-ci Please test |
|
@shahmishal Is there still something more to be tested here before merging? |
|
@rudkx yes, we need to update the website and few other scripts to make it work with |
Currently development snapshots for Linux are provided as
*.tar.gzfiles (Gzip compression). This patch switches the archive format to*.tar.xz(LZMA2 compression).There are lots of discussions out there that talk about why LZMA2 format might be preferred over Gzip so I'll omit them here.
For this project, the major benefit is an approximate 46% reduction in the source snapshots provided on Swift.org (from ~85MB to ~58MB). The trade-off is that it does add a little bit of time to the build, on my machine it amounts to another minute or so. Considering that the over-all build currently takes longer than an hour I would consider this addition "in-the-noise" and a fair trade off. Your experience and opinion may differ. Additionally another often cited negative about LZMA2 vs Gzip is compatibility. Both of the targeted Linux builds
tarutilities support LZMA2.My expectation is that the number of downloads is much much greater than than the number of people building from source and packaging. With that in mind this seems like an easy gain with little downside.