Build Pipeline V3 Wishlist

Owned by Matthew Lagoe
Last updated: Jan 27, 2014
2 min read

 

The current "V2" Python-based build pipeline is, in terms of structure, a 1:1 copy of the original (bat+sh+...) build pipeline.
Differences are mainly the removal of many hard coded directories into clearly separated configuration files, as well as stabilization fixes for ensuring all 3ds Max exports run without requiring manual interventions.
In it's current state all configuration files are simply in Python script, for development simplicity.

Wishlist

  • Configuration files should be in an easy to read format for external GUI support. JSON looks like a good candidate. (Using georges sheets allows to use existing gui for configuration.)
    • There must be a method for parenting configuration files to avoid duplicate configuration data.
    • A method for configuring post-configuration modifier scripts to run, to fill in macros, and to fill in configuration fields that can be gathered from data files such as sheets where possible.
  • Tools must be modified to allow passing a filelist, instead of requiring to copy files into a single folder.
  • Export and build will be merged into a single main script, with processes separated by category (misc/3d/leveldesign/etc) instead.
    • I disagree with this suggestion. I don't see the benefit in reduction of flexibility. If nothing else use export_build_install.py with ipj, epj, ipc and epc. If you want a variant that excludes the 'install' step then we can create an export_build.py.
      • RE: The distinction between export and build is for most processes in the pipeline just a matter of working order, and not relevant. A lot of export builds, a lot of build exports.
  • Improve methods for data inclusions across projects, to avoid requiring to configure certain folders twice.
  • A verify script which purely outputs the synchronization state of the build output.
  • All input and output must have metadata generated and associated with it, listing build input and output dependencies, and timestamps.
  • In the client_dev, folders such as data_common and gamedev which are currently copied 1:1 to w:\install\ by the pipeline, should be linked directly trough the cfg, and should not be copied.
  • A step that creates a distributable client (already have this) and server each in there own subfolder that can can just be zipped and ran
  • rather then use tga2dds.exe use a png2dds so map_panoply is unnecessary (saves a lot of build time and ~10gb space) (Panoply generates multiple tga files, this suggestion is invalid.)
    • I'm not sure why a 'png2dds' would save time and eliminate panoply?
  • use DXT1 rather then DXT5 (DXT5 is only needed when you need alpha transitions)
  • multithreaded (And on multiple machines!)
  • 3dsMax will be interfaced with using the COM Server interface. This allows us to monitor the progress.

Documentation

Pipeline