Michael's initial split notes: * Requirements: + split building is not mandatory + we still require straight-through builds for Win32 eg. + the split is broadly based on Kendy's layout here: + http://www.mail-archive.com/dev%40tools.openoffice.org/msg00786.html + we want to share as much as possible between distros, and shrink down-stream packagers's tweaks. + neither Debian nor RPM distros need accurate file lists, that is if we can split the package installation ourselves, we can just use wildcard patterns. * Implementation + the source is split by ooo-build/bin/src-pack2 + you need to autogen ooo-build with --with-split + we first configure the 'bootstrap' package (--with-piece) - with all the dependencies we require, and generate the LinuxIntelEnv.Set.sh that we will use for all subsequent builds. + this is lame: we could re-generate this per module of course - but it requires work and waste. + bootstrap installs all the patches, solenv, dmake etc. into /usr/lib/ooo-3/ solenv/ ooo-build/ solver/ + we apply patches from ooo-build/patches/dev300 with --pieces - this only applies fragments for top-level directories that exist (using bin/sloppypatch.pl) + we then build each subsequent module with bin/piece/build-generic and install-generic, having setup some pointers to /usr/lib/ooo-3/ + build-generic works by patching 'build.pl' to add a new --subdirs option: "build this lot & work out deps yourself" + before we build we do targetted clobbering of critical paths and environment variables in bin/piece/sys-setup - this overrides a number of key paths and includes to allow building from the system. + as we build, we try to build from two solvers: that inside the local package build tree, and also that installed in the system. We could avoid this by splitting the build into each top-level package, but this would give us too much fragmentation. + Having two solvers causes some interesting problems: + in particular the 'SOLARVER' (and co.) solver links now point to the system: /usr/lib/ooo-3/solver + unfortunately - many parts of the build, don't understand that a file generated inside that module will not be in the system solver, but in the local solver. + the piece-* patch set fixes these places. + To build you need to build things in this order: + bootstrap, ( ure | libs_extern | artwork ), libs_gui, libs_core, components, l10n-extras, filters ( impress | base | writer | calc | help ) + [ test building components / l10n in parallel ] + [ try move licenses somewhere else => might build l10n-extras in parallel with the rest??? ] + sample .spec files are in scratch/split/ooo3_*.spec + it is possible to add extra localization sources for each piece; just put it into the directory -/ooo-build/sdf; It is better to split it by language "any-name-.sdf For example: ooo300-m9-writer/ooo-build/sdf/writer-hu.sdf See also ooo-build/bin/ooo-sdf-split + the experimental sections can be enabled in bootstrap by the configure option --with-additional-sections More sections can be separated by comma ','. Aternatively, you might enable piece-specific experimental sections if you mention them in the file -/ooo-build/apply-additional-sections; More sections can be separated by comma ',' or by space " " or by new line. + it is possible to add/replace original diffs from the bootstrap package; just put diffs in the direcotry -/ooo-build/hotfixes the diff in the hotfixes dir overrrides the original diff of the same name; the other diff files in the hotfixes direcotry (with new names) are applied in the alphabetical order after all the regular diffs; + it is possible to modify the install process by the scripts: -/ooo-build/bin/post-upstream-install -/ooo-build/bin/post-install the first one is called after the upstream installer finishes; the second one is called at the very end after all the ooo-build-specific hacks + generated packages: + math from writer + 400kB big rpm + filters-optional from filters + gnome, kde, mono, pyuno, officebean, testtool + deps TODO: + FIXME: split filters into more subpackages? + split out binfilters? + per application? + FIXME: kde, gnome subpackages (useful because of dependencies) + vcl plugin in libs-gui + fpicker in libs-core + gconfbe1.uno.so, kdebe1.uno.so in shell in libs-core + libevoab2.so, libkabdrv1.so in connectivity in libs-core + FIXME: sdk, sdk-doc + still needed? + FIXME: check base package + are the database drivers needed for writer as well? + FIXME: check dependencies between the subpackages; make sure that it does not cause serious problems when an optional subpackage is not installed, search also officecfg below + FIXME: the system of inst-$piece, post-inst-$piece, file-list-$piece is a bit over-engineered (Petr's work); one piece-specific script should be enough and more clear; it should be done close the end (before fixing permissions) + rework inst-artwork to install icons into the installed sovler + merge file-list-* and post-inst-* + FIXME: naming: + rename install-l10n-helper to install-noarch-helper? + FIXME: source split: + generate the hacky setup_native_packinfo-ooo300-m3.tar.bz2 + split help from l10n + can and should be built separately + move main_transform.xsl to ll0n-extras + move hid.lst to ??? (where the testtool is being built) + rename l10n to l10n-extras (also the corresponding file-list-l10n) + move officecfg to ure? we need to register the optional peices only when they are installed, for example: basis3.0/share/registry/modules/org/openoffice/Office/Common/Common-base.xcu adds the Database menu entry (IMHO) + FIXME: reduce l10n-extras-devel + it includes many .zip archives but the files are already installed + FIXME: the all_filelist.txt we use in copyexcept should list both source & dest files, and handle links, by adding symlinks to the system-installed library for lib symlinks eg. solver/lib/libfoo.so -> ../../program/libfoo.so.1.2.3 etc. + this should shrink the -devel packages substantially. + FIXME: the services building is -horrific- and needs fixing. + FIXME: where libraries are symlinked - we install copies of these into the system sometimes multiple times, which bloats our -devel packages horribly & unnecessarily. + FIXME: the common helpcontent2 files (main_transform.xsl idxcaption.xsl idxcontent.xsl) should be passed to the main package via the help-devel package; it is currently done via ooo-build/src + we install all the artwork into the system: + yet really it would be far better to install just the .zip archives, and work from them [ perhaps with a manifest for tools to grok ? ] + then we could drop the rsc2 patch to not check for icons' existence. + drop a 'Makefile' from the system in each top-level as we unpack it, that will do all the build goodness (?) + the piece-* patches need re-factoring, adding common env. variables to be re-used everywhere & moved up-stream, eg. URESOLARBINDIR instead of SOLARBINDIR etc. + make the "straight through" build look much more like the normal build. --- older notes --- Build splitting notes: + we need the same environment, and we need to link against a given path instead of the solver. + so for 'sw' we need: + isolate it. + get include files out - into separate place. + do we need IDL ? + libraries from: + /opt/OOInstall/basis3.0/program * Major tasks: + incremental pieces: + hack bits out of scp2 as we go: + #ifndef EXTERNAL_SW + #endif // etc. + install static libraries too [somewhere] ? + basically we want 'deliver' really ... + can we create a boot-strapping OO.o without much pain otherwise: just bastardize make_installer.pl ? + how much does it do that is really useful ? + building component registry / rdbs (?) + need to do that on rpm install anyway ? + do it in one big rpm at the end ? + have file-install lists per module (d.lst) ... (?) + tag files "devel" vs. "non-devel" + or have some black-list in a 'deliver' script ? + possible deliver heuristics: + all libraries -> installed system + attempt to register each of them in registry.rdb + [ what RPM file contains that rdb ? ] + can we not do that at the end ? + meta 'OpenOffice' package that does registration. + make make_installer run against the final system + all uiconfig* -> install + all layout*.zip -> install + have a path mapping: + from solver -> inst image. + split the scp2 - move it into each module ... [!?] ... + Can we bootstrap from 'deliver' !? + *or* - add tags to the scp - annotating where it comes from ? + or run the make_installer in a minimal way that just installs what is there ;-) + [ we would need to accelerate that script ... ] cannot find -luno_cppuhelpergcc3 + we need symlinks for: + /opt/OOInstall/ure/lib/libuno_cppuhelpergcc3.so.3 -> .so [etc.] + a -devel RPM for this. + *or* ... something else (?) + just whack the symlinks in ure/lib itself ... (?) for a in *.so.*; do ln -s $a `echo $a | sed 's/\.[0-9]*$//'`; done Common pieces we need in the "finish up" phase: + icon: ilst files + res: files [ translations ] + make configure do that clobbering ? + create a dummy Makefile to wrap the beastie. + get on with it ;-) + Translation [! -urgh- ] + ideas + duplicate all makefile.mk's into a separate lang-package + duplicate all .src files [etc.] into there + FIXME: if we patch things; what then ? + we need to be the canonical 'master' really. + include translation tools in there [ self contained ] + + Allow parallel translation builds + insert dummy translation rules into the makefiles for writer etc. just 'touch' the files. + [ or? ] + Translation is spread all over the code + => we need two modes + "compile mode" and "translate mode" + do these exist already ? + or do we cheat ? + just whack the translation files into the -devel package ? + and copy from that (?) Resolved: + copy the translations into a known place in the -devel package and collate them later into the main OO.o binary. + allows developer to add new strings easily etc. + adding new / individual translations perhaps a big pain though. + same for ilst files [etc.] + hack this into 'deliver' + build the 'solver' much in the same way. + just minus the installable files. + sort out the ooo-build wrapper ... + urgh; + make 'sw' build standalone nicely ... + fix the wrapper later ? + 'postprocess' ... for a in ../HEAD/src/dev300-m19*.tar.bz2; do tar -xjf $a; done + configure.in: + added "--with-split" ... --with-installed-ooo-dirname=%ooo_home: ooo_home => "ooo-2.0" from "setup.in": OOINSTBASE=@libdir@/@OOOINSTALLDIRNAME@ + whack the solver in: /usr/lib/ooo-2.0/solver/ + prune down the things that are simply not used ... * ooo-build: how much hassle do we need ? + add a "--with-devel-path" and default to /usr/lib/ooo-2.0/solver + we won't need to do -so- much work in an 'inferior' package (?) + all the configure options will be set ahead of time. + --with-devel-path= - will set all the options (?) - [ in the #1st analysis at least ;-] ** FIXME: + we need our 'build logic' split up and used in lots of different places. + "unpack" needs splitting into lots of pieces. + "ooinstall" likewise + can we create a set of scripts: one per piece, and just run them in sequence ? build.pl: + from 'BuildAll' ? - goodness starts (?) => GetParentDeps () ... => $build_all_cont ... [!] + PickPrjToBuild ... -> FindIdepPrj + all comes from the build hash ... + build_multiprocessing Questions: + should we try a different approach + "solver" is the system 'solver' + deliver [ etc. ] tweak it ... + my $build_sosl = $ENV{'BUILD_SOSL'}; my $common_outdir = $ENV{'COMMON_OUTDIR'}; my $inpath = $ENV{'INPATH'}; my $solarversion = $ENV{'SOLARVERSION'}; my $updater = $ENV{'UPDATER'}; my $updminor = $ENV{'UPDMINOR'}; my $work_stamp = $ENV{'WORK_STAMP'}; install to 'SOLARVERSION' ? * deliver is the only thing that copies output anywhere; => we need to fool deliver. => we need to ensure that our compile flags [etc.] include the local intermediate / non-solver. * Debian: doesn't need file lists ... * RPM doesn't really need file-lists either * FIXME: + is transex3/source/lngex - what we want ? + do we even want scp2 ? + rather a waste of time surely ? + why not just bin it ? :-) exciting TODO: + %dir %ooo_prefix/%ooo_home/basis3.0 %dir %ooo_prefix/%ooo_home/LICENSE %dir %ooo_prefix/%ooo_home/LICENSE.html %dir %ooo_prefix/%ooo_home/README %dir %ooo_prefix/%ooo_home/README.html %dir %ooo_prefix/%ooo_home/readmes %dir %ooo_prefix/%ooo_home/licenses %dir %ooo_prefix/%ooo_home/program %dir %ooo_prefix/%ooo_home/share %dir %ooo_prefix/%ooo_home/ure %dir %ooo_prefix/%ooo_home/THIRDPARTYLICENSEREADME.html %ooo_prefix/%ooo_home/basis3.0/* %ooo_prefix/%ooo_home/LICENSE/* %ooo_prefix/%ooo_home/LICENSE.html/* %ooo_prefix/%ooo_home/README/* %ooo_prefix/%ooo_home/README.html/* %ooo_prefix/%ooo_home/readmes/* %ooo_prefix/%ooo_home/licenses/* %ooo_prefix/%ooo_home/program/* %ooo_prefix/%ooo_home/share/* %ooo_prefix/%ooo_home/ure/* %ooo_prefix/%ooo_home/THIRDPARTYLICENSEREADME.html/* make_installer.pl [!] ... can't mangle include path ... + need a different include path for certain control files [!?] filters - urgh; tons of flaking ANT / classpath bits ... find BUILD/dev300-m25-filters -name 'build.xml' * Final missing pieces: + services.rdb + needed *both* basis3.0 and ure ... + can we hack the installer, to build one services.rdb per module ? [ might be easier ! ]. * FIXME: + done in the cleanup section ? + packimages [etc.] * FIXME: + startup - and moaning about missing Resources ... + means missing bootstraprc files (etc. in both program/ and basis3.0/program/) ./instsetoo_native/util/OpenOffice/profiles/en-US/gid_Profile_Version_Ini_Basis/versionrc ./instsetoo_native/util/OpenOffice/profiles/en-US/gid_Profile_Version_Ini_Ure/versionrc ./instsetoo_native/util/OpenOffice/profiles/en-US/gid_Brand_Profile_Version_Ini/versionrc ** FIXME: + these files are generated -entirely- from the scp ... + "ProfileItem" and "ConfigurationItems" + in scriptitems.pm and profiles::create_profiles (?) ... ? + wow :-) + export OOO_ALLOW_PROFILE=1 ... + add an --inhibit-profiles flag (?) + use it everywhere - except for the last stage ... (?) ** FIXME: + we have 55Mb -> 18Mb [compressed] of un-necessary crud in /usr/lib/ooo3/solver/lib/* + the installer script needs tweaking ... + [ some tweaks - just bin the binfilter .a files ? ] + accelerating patch application ... + patched_files + Petr's brain dump (2008-11-07) + FIXME: merge with the above: The known functional bugs (from my list, need to fix most of them for openSUSE-11.1 withing next 2-3 weeks): + pyuno components are still not registered + mono stuff is not packaged correctly + localized strings are not updated from extra sources + the user configuration includes symlinks to /usr/share/ooo3 instead of that real files; it might be a feature but it might also cause problems in the future (the symlinks are there because I build more stuff noarch now and there are the compat symlinks between /usr/lib and /usr/share) + missing OOo-sdk compat stuff (to build voiko..., bnc#428403) More functional bugs (not sure if caused by the split build): + mailmerge does not work + Suse-puzzler.xls works only partly; e.h. "Sneaky Peak" or "About" works but the "Shuffle" does nohing. Also, I can't move the pieces by mouse. The known build problems: + some more missing hacks from ooo-build/bin/package-ooo and older OOo.spec (hight priority) + allow to change build of pieces without rebuilding bootstrap package (high priority): + hotfixes directory for additional diffs + optional postinstall script, ... + better merge the non-split/split ooo-build scripts (setup vs. piece/sys-setup, install-mono, localize-ooo, ...); we need to reduce the cut&paste + allow configure/make/make install process: + add configure for every piece? + add makefile target for every piece + ... Clean up & upstreaming (avoid hacks): + avoid hack with extra setup_native_packinfo-ooo300-m4.tar.bz2 sources + split scp2, officecfg per module + split configure per piece + review all the piece-*.diffs (I am not familiar with them yet) Enf of Petr's brain dump (2008-11-07)