Developers memo (original) (raw)

KS compiler is written in Scala languageand thus uses SBT for building. It can be compiled in one of 2 ways, each of which can be packaged in one of several ways:

Prerequisites

To build the compiler, one generally needs just these two things:

For Debian/Ubuntu, this should be enough to download every prerequisite (JRE is pulled automatically by sbt):

echo "deb https://repo.scala-sbt.org/scalasbt/debian all main" | sudo tee /etc/apt/sources.list.d/sbt.list
echo "deb https://repo.scala-sbt.org/scalasbt/debian /" | sudo tee /etc/apt/sources.list.d/sbt_old.list
curl -sL "https://keyserver.ubuntu.com/pks/lookup?op=get&search=0x2EE0EA64E40A89B84B2DF73499E82A75642AC823" | sudo apt-key add
sudo apt-get update
sudo apt-get install sbt

For RedHat-based systems, use the following:

curl -L https://www.scala-sbt.org/sbt-rpm.repo > sbt-rpm.repo
sudo mv sbt-rpm.repo /etc/yum.repos.d/
sudo yum install sbt

Manual installation:

Note You don’t need whole Java Development Kit (JDK), unless you want to test & develop for Java target — just JRE is enough for KS compiler.

All other dependencies (including Scala runtimes, compilers, etc) required for the compiler are downloaded and installed by sbt automatically. It shouldn’t really matter even which version of sbt you use for bootstrap, as sbt will pull relevant sbt update packages automatically as well.

Configuring proxy

Note, that if you behind proxy, you need to run sbt with flags

  -Dhttp.proxyHost=<your proxy server>
  -Dhttp.proxyPort=<your proxy server port>
  -Dhttps.proxyHost=<your proxy server>
  -Dhttps.proxyPort=<your proxy server port>

For example

  sbt -Dhttp.proxyHost=proxy.com -Dhttp.proxyPort=3128 -Dhttps.proxyHost=proxy.com -Dhttps.proxyPort=3128

Unfortunately, sbt doesn’t understand http(s)_proxy environment variable properly, if it contains address of proxy server in host:port format, so flags is necessary.

This flags needed only on first run or when you want to check and upgrade dependencies. After first sbtrun all dependencies will be downloaded and access to the internet not required anymore.

Building for JVM

We use sbt-native-packager to build deployable formats. Note that in order to perform the following builds, one must be in the kaitai_struct/compiler directory.

Building an universal (.zip) package

  1. sbt compilerJVM/universal:packageBin
  2. Get result in jvm/target/universal/kaitai-struct-compiler-*.zip

Building Debian package

  1. Install prerequisites:
    • On Debian/Ubuntu host: sudo -i apt-get install dpkg dpkg-sig dpkg-dev lintian fakeroot
    • On RedHat host: sudo dnf install dpkg fakeroot
  2. sbt compilerJVM/debian:packageBin
  3. Get result in jvm/target/kaitai-struct-compiler_*_all.deb

Building RedHat package

  1. Install prerequisites:
    • On RedHat host: sudo dnf install rpm rpm-build
  2. sbt compilerJVM/rpm:packageBin
  3. Get result in jvm/target/rpm/RPMS/noarch/kaitai-struct-compiler-*.noarch.rpm

Building Windows package

  1. Install WIX
  2. sbt compilerJVM/windows:packageBin
  3. Get result in jvm/target/windows/kaitai-struct-compiler.msi
  4. Rename to add version to kaitai-struct-compiler-$VERSION.msi

Building for JavaScript platform

Building to JavaScript platform is done using a Scala.js project. Note that it uses a somewhat different set of dependencies, as they must actually be JavaScript libraries, not Java jars.

  1. Run sbt fastOptJS
  2. Get result in js/target/scala-2.11/kaitai-struct-compiler-fastopt.js
  3. Use this JavaScript file on a website

Publishing a new version

  1. Choose a new version number (WIX imposes harsh requirements for version to look like x.x.x.x) and update it in build.sbt,version := …​, commit
  2. Prepare an entry in RELEASE_NOTES.md, commit
  3. Create version tag:
    • git tag $VERSION
    • git push --tags
  4. Update main repository
  5. Create new version at:
  6. Upload:
  7. Publish them all

Publishing new version to oss.sonatype.org

  1. Verify that one has OSS Sonatype login/password for iokaitai org.
  2. Preliminary setup (needs to be done once per machine — verified for sbt 1.1)
    • Enable the sbt-pgp plugin globally by adding this line to ~/.sbt/1.0/plugins/gpg.sbt:
    addSbtPlugin("com.github.sbt" % "sbt-pgp" % "2.1.2")
    • Set up credentials: create $HOME/.sbt/.credentials with the following contents (replacing XXX with username and password):
      realm=Sonatype Nexus Repository Manager
      host=oss.sonatype.org
      user=XXX
      password=XXX
    • Set up $HOME/.sbt/1.0/plugins/credentials.sbt with the following contents:
    credentials += Credentials(Path.userHome / ".sbt" / ".credentials")
    • For publishing the Java runtime library: set up ~/.m2/settings.xml with the following contents:
    <settings>  
  <servers>  
    <server>  
      <id>ossrh</id>  
      <username>XXX</username>  
      <password>XXX</password>  
    </server>  
  </servers>  
</settings>
    • Make sure GPG keys are present
  3. sbt publishSigned
  4. Go to https://oss.sonatype.org/#stagingRepositories
  5. Continue to follow Java runtime publishing instructions