Compilare in parallelo con i clusters.

Guides, tutorials & docs
Post Reply
User avatar
Roberto_65
Collaboratore
Collaboratore
Posts: 516
Joined: 6 December 2007, 23:56
OpenMandriva: 2009.1
Kernel: i686 x86_64
Desktop: Gnome Xfce4
Location: Triangolo delle Bermude
Contact:

Compilare in parallelo con i clusters.

Post by Roberto_65 »

Il Clustering è la modalità che offre Linux di unire più computer fra loro attraverso le schede di rete in modo da creae un solo ed unico supercomputer composto da N CPU.

La compilazione e` spesso un'attivita` molto lunga, in particolare per progetti di una certa grandezza come il kernel Linux o le interfacce grafiche come Gnome o Kde.
Su macchine multiprocessore o con architetture particolari (Hyperthreading o multicore) potrebbe essere utile compilare parti del codice in parallelo.
Potrebbe essere interessante sfruttare un cluster (anche di ridotte dimensioni) per la compilazione di un progetto molto voluminoso, in quanto i processi di compilazione potrebbero essere migrati verso i vari nodi del cluster, per poi effettuare il linking degli oggetti sulla macchina che ha lanciato il comando. Inoltre e` anche possibile usare cluster mosix, in quanto la memoria non e` condivisa tra i job.

L'utilizzo di un cluster, sebbene di elevato interesse a livello implementativo, ha scarsa utilita` a livello pratico: la compilazione di un applicazione, per voluminosa che sia (anzi, a maggior ragione...) avviene molto di rado; e` dunque preferibile dedicare un cluster per il calcolo e l'elaborazione, invece che alla compilazione. Esiste comunque un progetto di un compilatore parallelo, distcc (http://distcc.samba.org)

Il pacchetto DISTCC
Quindi scarichiamo il programma DISTCC da http://distcc.samba.org/download.html
(se qualcuno ha tempo lo potrebbe pacchettizzare in RPM per Mandriva. Io l'ho già fatto per Ubuntu e funziona)

1. For each machine, download distcc unpack, and do
./configure
make
make install
2. On each of the servers, run distccd --daemon, with --allow options to restrict access.
3. Put the names of the servers in your environment:
export DISTCC_HOSTS='localhost red green blue'
4. Build!
cd ~/work/myproject; make -j8 CC=distcc

distcc is a program to distribute builds of C, C++, Objective C or Objective C++ code across several machines on a network. distcc should always generate the same results as a local build, is simple to install and use, and is usually much faster than a local compile.

distcc does not require all machines to share a filesystem, have synchronized clocks, or to have the same libraries or header files installed. They can even have different processors or operating systems, if cross-compilers are installed.

"Just wanted to drop you a line to say that we are now using distcc at work and it is excellent. We have a rack of opteron machines that we use for computational tasks and we are now using them as a compile farm to compile our ~1MLOC C++ tree (which can take an hour on a single CPU to recompile if we change certain header files). We tried using Sun's grid engine to do this (we already use it to schedule our computational jobs), but the combination of its polled operation and the overhead of NFS led to little improvement (and greatly stressed our network). With distcc compile times are way down and my productivity has greatly improved; the two best features for me are its low impact on the network and its simplicity.
  • distcc 2.18.3 i486-pc-linux-gnu (protocols 1 and 2) (default port 3632)
    built Aug 5 2008 01:26:08
    Copyright (C) 2002, 2003, 2004 by Martin Pool.
    Includes miniLZO (C) 1996-2002 by Markus Franz Xaver Johannes Oberhumer.

    distcc comes with ABSOLUTELY NO WARRANTY. distcc is free software, and
    you may use, modify and redistribute it under the terms of the GNU
    General Public License version 2 or later.

    Built with Zeroconf support.

    Usage:
    distcc [COMPILER] [compile options] -o OBJECT -c SOURCE
    distcc --help

    Options:
    COMPILER defaults to "cc"
    --help explain usage and exit
    --version show version and exit
    --show-hosts show host list and exit
    -j calculate the concurrency level from
    the host list.

    Environment variables:
    See the manual page for a complete list.
    DISTCC_VERBOSE=1 give debug messages
    DISTCC_LOG send messages to file, not stderr
    DISTCC_SSH command to run to open SSH connections
    DISTCC_DIR directory for host list and locks

    Server specification:
    A list of servers is taken from the environment variable $DISTCC_HOSTS, or
    $DISTCC_DIR/hosts, or ~/.distcc/hosts, or /etc/distcc/hosts.
    Each host can be given in any of these forms, see the manual for details:

    localhost run in place
    HOST TCP connection, port 3632
    HOST:PORT TCP connection, specified port
    @HOST SSH connection
    USER@HOST SSH connection to specified host

    distcc distributes compilation jobs across volunteer machines running
    distccd. Jobs that cannot be distributed, such as linking or
    preprocessing are run locally. distcc should be used with make's -jN
    option to execute in parallel on several machines.
Attachments
distccmon-gnome-2003-09-23-01.png
distccmon-gnome-2003-09-23-01.png (17.72 KiB) Viewed 2720 times
Roberto_65
Packager delle MIB-Live
Il creatore delle MIB-Live
L'inventore di MIB-LiveToFlash
Triangolo delle Bermude http://www.sitohd.com/siti/3209

User avatar
SymbianFlo
Utente sostenitore
Utente sostenitore
Posts: 1493
Joined: 7 December 2007, 20:07
OpenMandriva: 2010.1 x86_64
Kernel: 2.6.33.5-nrj-69mib
Desktop: kde4.5.rc1
Location: Pordenone
Contact:

Re: Compilare in parallelo con i clusters.

Post by SymbianFlo »

hmmm interessante quasi quasi li frego una core alla femmina ghghghghghghg
:D ciao ciao SymbianFlo :D
Image
Linus Torvalds only has to enter a room, and every Windows computer in it segfaults instantly.
http://video.linuxfoundation.org/video/1057

User avatar
Roberto_65
Collaboratore
Collaboratore
Posts: 516
Joined: 6 December 2007, 23:56
OpenMandriva: 2009.1
Kernel: i686 x86_64
Desktop: Gnome Xfce4
Location: Triangolo delle Bermude
Contact:

Re: Compilare in parallelo con i clusters.

Post by Roberto_65 »

Se qualcuno possa fare una guida ben dettagliata dettata dalla propria esperienza sarebbe meglio !!!!

Ciao
Roberto_65
Packager delle MIB-Live
Il creatore delle MIB-Live
L'inventore di MIB-LiveToFlash
Triangolo delle Bermude http://www.sitohd.com/siti/3209

Post Reply