Skip to content

[build-script] Reduce the size of development snapshots #801

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from

Conversation

RLovelett
Copy link
Contributor

Currently development snapshots for Linux are provided as *.tar.gz files (Gzip compression). This patch switches the archive format to *.tar.xz (LZMA2 compression).

There are lots of discussions out there that talk about why LZMA2 format might be preferred over Gzip so I'll omit them here.

For this project, the major benefit is an approximate 46% reduction in the source snapshots provided on Swift.org (from ~85MB to ~58MB). The trade-off is that it does add a little bit of time to the build, on my machine it amounts to another minute or so. Considering that the over-all build currently takes longer than an hour I would consider this addition "in-the-noise" and a fair trade off. Your experience and opinion may differ. Additionally another often cited negative about LZMA2 vs Gzip is compatibility. Both of the targeted Linux builds tar utilities support LZMA2.

My expectation is that the number of downloads is much much greater than than the number of people building from source and packaging. With that in mind this seems like an easy gain with little downside.

@@ -2249,10 +2249,10 @@ if [[ "${INSTALLABLE_PACKAGE}" ]] ; then
fi

(cd "${INSTALL_DESTDIR}" &&
tar -c -z -f "${INSTALLABLE_PACKAGE}" "${TOOLCHAIN_PREFIX/#\/}")
tar -c -J -f "${INSTALLABLE_PACKAGE}" "${TOOLCHAIN_PREFIX/#\/}")
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a branch specific to OS X. tar on OS X does not support the -J flag.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My version of OS X does. I don't think my version of tar is from Homebrew; but perhaps it is? I'm fine with removing it if that is your desire. Just thought I'd point it out.

I did at least test the command on my install OS X El Capitan 10.11.2 (15C50) and it supported the -J flag.

$ tar --help
tar(bsdtar): manipulate archive files
First option must be a mode specifier:
  -c Create  -r Add/Replace  -t List  -u Update  -x Extract
Common Options:
  -b #  Use # 512-byte records per I/O block
  -f <filename>  Location of archive
  -v    Verbose
  -w    Interactive
Create: tar -c [options] [<file> | <dir> | @<archive> | -C <dir> ]
  <file>, <dir>  add these items to archive
  -z, -j, -J, --lzma  Compress archive with gzip/bzip2/xz/lzma
  --format {ustar|pax|cpio|shar}  Select archive format
  --exclude <pattern>  Skip files that match pattern
  -C <dir>  Change to <dir> before processing remaining files
  @<archive>  Add entries from <archive> to output
List: tar -t [options] [<patterns>]
  <patterns>  If specified, list only entries that match
Extract: tar -x [options] [<patterns>]
  <patterns>  If specified, extract only entries that match
  -k    Keep (don't overwrite) existing files
  -m    Don't restore modification times
  -O    Write entries to stdout, don't restore to disk
  -p    Restore permissions (including ACLs, owner, file flags)
bsdtar 2.8.3 - libarchive 2.8.3

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Indeed, it does. My bad -- I was checking the man page.

@gribozavr
Copy link
Contributor

Is it supported on RHEL 7 / CentOS 7?

@gribozavr
Copy link
Contributor

I don't have objections to this change, but we need to coordinate with @shahmishal to update other automation scripts that handle these packages.

@RLovelett
Copy link
Contributor Author

(I'm trying to get a CentOS 7 VM image now to test on it)

Frankly I don't have a strong opinion either way. If it breaks too many processes or things that I cannot see from the outside then it won't break my heart to see this closed.

I use it on my own local daily builds just to keep the file storage down. The patch is small enough for me to maintain myself without it being subsumed into the mainline. I thought the community may benefit from the patch so I submitted it.

@gribozavr
Copy link
Contributor

If the LZMA2 compression is supported everywhere, then using it does not "break" processes, it just requires changes to them in places that somehow assume *.tar.gz filenames or gzip compression.

@RLovelett
Copy link
Contributor Author

I have now built a CentOS 7.1 VM and can confirm that it also supports LZMA2 compression.

@gribozavr
Copy link
Contributor

@RLovelett Sounds great, thank you for checking. Let's coordinate landing this change with @shahmishal after the break.

Currently development snapshots for Linux are provided as `*.tar.gz`
files (Gzip compression). This patch switches the archive format to
`*.tar.xz` (LZMA2 compression).

There are lots of discussions out there that talk about why LZMA2 format
might be preferred over Gzip so I'll omit them here.

For this project, the major benefit is an [approximate 46%
reduction](http://www.wolframalpha.com/input/?i=percent+difference&f1=60306748&f=PercentDifference.number1_60306748&f2=88476335&f=PercentDifference.number2_88476335)
in the source snapshots provided on Swift.org (from ~85MB to ~58MB). The
trade-off is that it does add a little bit of time to the build, on my
machine it amounts to another minute or so. Considering that the
over-all build currently takes longer than an hour I would consider this
addition "in-the-noise" and a fair trade off. Your experience and
opinion may differ. Additionally another often cited negative about
LZMA2 vs Gzip is compatibility. Both of the targeted Linux builds `tar`
utilities support LZMA2.

My expectation is that the number of downloads is much much greater than
than the number of people building from source and packaging. With that
in mind this seems like an easy gain with little downside.
@DougGregor
Copy link
Member

@shahmishal , do you have concerns or should we go ahead with this?

@shahmishal
Copy link
Member

Before we merge this, will need to make sure our backend (code sign system and swift.org) is ready to handle it.

@shahmishal
Copy link
Member

@swift-ci Please test

@rudkx
Copy link
Contributor

rudkx commented Jan 12, 2017

@shahmishal Is there still something more to be tested here before merging?

@shahmishal
Copy link
Member

@rudkx yes, we need to update the website and few other scripts to make it work with *.tar.xz

@RLovelett RLovelett closed this Jul 5, 2017
@RLovelett RLovelett deleted the xz-vs-gz branch April 15, 2020 02:16
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants