How to build hadoop source code using ant & ivy (hadoop release-1.0.4)

Download hadoop source code

Use svn anonymous checkout to get hadoop source code

create folder to checkout source code, for example:
mkdir -p /scratch/rajiv/softwares/hadoop/source


Now checkout from svn repository.

$svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/ release-1.0.4

svn: OPTIONS of 'http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4': could not connect to server (http://svn.apache.org)

Got above error initially as I am behind proxy server.

Edit ~/.subversion/servers file and un-comment below lines and provide your proxy server hostname(preferably fully qualified hostname) and port. If you don't know the proxy serve name, just run "wget google.com" from terminal, it will print proxy server hostname.

Make sure you edit these properties under "[global]" section. Same properties are present under "group".

But modifying them will not have any effect. You will get same exception as above.

#http-proxy-host=my.proxy.server.name.here#http-proxy-port=80#http-compression = no

Now run svn checkout again

$svn checkout http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/ release-1.0.4

Now the hadoop source code is checkout out under current folder.

$ls hadoop-common-1.0.4

Build hadoop source code
Hadoop 1.0.4 source doesn't have maven project defined.
There is no pom.xml present and trying to build using maven will fail with below exception.
[ERROR] The goal you specified requires a project to execute but there is no POM in this directory (/scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4). Please verify you invoked Maven from the correct directory. -> [Help 1]

Maven project was available till hadoop release-0.23.7.

Then versions from release-0.3.0  to release-0.9.2 ant is used.

Then version from release-1.0.0 to release-1.2.0-rc1 ant and ivy are used(noticed that during build, ivy downloads maven2 artifacts).

Now from hadoop 2.0 on-wards ant and ivy are removed and only maven project is present.

 
Refer 

http://svn.apache.org/repos/asf/hadoop/common/tags/release-0.23.7/
 
http://svn.apache.org/repos/asf/hadoop/common/tags/release-1.0.4/

http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.0.1-alpha/


Install ant and Ivy


Download and extract ivy
 wget http://apache.osuosl.org//ant/ivy/2.3.0/apache-ivy-2.3.0-bin.tar.gz


download and extract ant
 
wget http://archive.apache.org/dist/ant/binaries/apache-ant-1.8.4-bin.tar.gz
$ant jar 


Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/tools/ant/launch/Launcher
Caused by: java.lang.ClassNotFoundException: org.apache.tools.ant.launch.Launcher
        at java.net.URLClassLoader$1.run(URLClassLoader.java:202)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:190)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:306)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:301)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:247)
Could not find the main class: org.apache.tools.ant.launch.Launcher.  Program will exit


set JAVA_HOME and ANT_HOME to fix above error.


$ setenv JAVA_HOME
/scratch/rajiv/softwares/hadoop/jdk
$ setenv ANT_HOME /scratch/rajiv/softwares/hadoop/ant


$/scratch/rajiv/softwares/hadoop/ant/bin/ant jar 
Buildfile: /scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4/build.xml

clover.setup:

clover.info:
     [echo]
     [echo]      Clover not found. Code coverage reports disabled.
     [echo]

clover:

ivy-download:
      [get] Getting: http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar
      [get] To: /scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4/ivy/ivy-2.1.0.jar
      [get] Error getting http://repo2.maven.org/maven2/org/apache/ivy/ivy/2.1.0/ivy-2.1.0.jar to /scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4/ivy/ivy-2.1.0.jar

BUILD FAILED
/scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4/build.xml:2419: java.net.NoRouteToHostException: No route to host
        at java.net.PlainSocketImpl.socketConnect(Native Method)
        at java.net.PlainSocketImpl.doConnect(PlainSocketImpl.java:351)
        at java.net.PlainSocketImpl.connectToAddress(PlainSocketImpl.java:213)
        at java.net.PlainSocketImpl.connect(PlainSocketImpl.java:200)
        at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:366)
        at java.net.Socket.connect(Socket.java:529)
        at java.net.Socket.connect(Socket.java:478)
        at sun.net.NetworkClient.doConnect(NetworkClient.java:163)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:388)
        at sun.net.www.http.HttpClient.openServer(HttpClient.java:523)
        at sun.net.www.http.HttpClient.(HttpClient.java:227)
        at sun.net.www.http.HttpClient.New(HttpClient.java:300)
        at sun.net.www.http.HttpClient.New(HttpClient.java:317)
        at sun.net.www.protocol.http.HttpURLConnection.getNewHttpClient(HttpURLConnection.java:970)
        at sun.net.www.protocol.http.HttpURLConnection.plainConnect(HttpURLConnection.java:911)
        at sun.net.www.protocol.http.HttpURLConnection.connect(HttpURLConnection.java:836)
        at org.apache.tools.ant.taskdefs.Get$GetThread.openConnection(Get.java:660)
        at org.apache.tools.ant.taskdefs.Get$GetThread.get(Get.java:579)
        at org.apache.tools.ant.taskdefs.Get$GetThread.run(Get.java:569)

Total time: 10 seconds
[rajiv@myhostname hadoop-common-1.0.4]$







ant is not able to get ivy jars. Pass http proxy host and port as -D arguments to ant to fix this.







$ /scratch/rajiv/softwares/hadoop/ant/bin/ant jar -Dhttp.proxyHost=your-proxy-server-name-here -Dhttp.proxyPort=80









This also doesn't work





set  ANT_OPTS



 setenv ANT_OPTS "-Dhttp.proxyHost=your-proxy-server-name-here -Dhttp.proxyPort=80"



$ /scratch/rajiv/softwares/hadoop/ant/bin/ant jar 









cd to hadoop source code folder, make sure build.xml file present under this folder:

 

$cd /scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4

run ant:

$/scratch/rajiv/softwares/hadoop/ant/bin/ant jar

sample output (trimmed), refer this link for complete build output log

/scratch/rajiv/softwares/hadoop/ant/bin/ant jar Buildfile: /scratch/rajiv/softwares/hadoop/source/hadoop-common-1.0.4/build.xml

clover.setup:

clover.info:
[echo]
[echo] Clover not found. Code coverage reports disabled.
[echo]
 ........................... 
 

BUILD SUCCESSFUL
Total time: 8 minutes 34 seconds
[rajiv@myhostname hadoop-common-1.0.4]$