User Tools

Site Tools


linux_hadoop_minimal_installation_instructions

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision
Previous revision
linux_hadoop_minimal_installation_instructions [2019/06/11 16:43]
deadline more tweaks
linux_hadoop_minimal_installation_instructions [2020/05/21 18:46] (current)
deadline
Line 5: Line 5:
 **Email:** deadline(you know what goes here)basement-supercomputing.com **Email:** deadline(you know what goes here)basement-supercomputing.com
  
-Unless otherwise noted, all course content, notes, and examples are +**Unless otherwise noted, all course content, notes, and examples are 
-(c) Copyright Basement Supercomputing 2019, All rights reserved.+(c) Copyright Basement Supercomputing 2019, All rights reserved.**
  
 ====What Is This?==== ====What Is This?====
Line 16: Line 16:
   * [[https://www.safaribooksonline.com/search/?query=Hands-on%20Introduction%20to%20Apache%20Hadoop%20and%20Spark%20Programming&field=title|Hands-on Introduction to Apache Hadoop and Spark Programming]]   * [[https://www.safaribooksonline.com/search/?query=Hands-on%20Introduction%20to%20Apache%20Hadoop%20and%20Spark%20Programming&field=title|Hands-on Introduction to Apache Hadoop and Spark Programming]]
  
-It can also be used for the examples provided in the companion on-line+It can also be used for the [[https://www.clustermonkey.net/download/LiveLessons/Hadoop_Fundamentals/|examples]] provided in the companion on-line
 video tutorial (14+ hours): video tutorial (14+ hours):
  
Line 37: Line 37:
  
   wget https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Hands_On_Hadoop_Spark-V1.5.tgz   wget https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Hands_On_Hadoop_Spark-V1.5.tgz
-  tar xvzf Hands_On_Hadoop_Spark-V1.5.tgz+  tar xvzf Hands_On_Hadoop_Spark-V1.5.1.tgz
  
 Similarly, for the "Linux Command Line" course (do this within the VM) Similarly, for the "Linux Command Line" course (do this within the VM)
Line 92: Line 92:
 **Step 3:** Make sure hardware virtualization is enabled in your BIOS. **Step 3:** Make sure hardware virtualization is enabled in your BIOS.
  
-**Step 4:** Download  the https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Linux-Hadoop-Minimal-0.42.ova image and load into VirtualBox. (NOTE newer version may be available.)+**Step 4:** Download the https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Linux-Hadoop-Minimal-0.42.ova image and load it into VirtualBox. (NOTE newer version may be available.) 
  
 **Step 5:** Start the VM. All the essential Hadoop service should be started automatically. **Step 5:** Start the VM. All the essential Hadoop service should be started automatically.
Line 286: Line 287:
 and describe the situation. and describe the situation.
  
-When the VM is stopped (see belwo) with ''poweroff'' or restarted with ''reboot'' commands, a script called ''/sbin/halt.local'' shuts down all the daemons. +When the VM is stopped (see below) with ''poweroff'' or restarted with ''reboot'' commands, a script called ''/sbin/halt.local'' shuts down all the daemons. 
  
 ====Stopping the VM==== ====Stopping the VM====
Line 308: Line 309:
 These issues have been addressed in the current version of the VM. Please use the lasted VM and you can avoid these issues. These issues have been addressed in the current version of the VM. Please use the lasted VM and you can avoid these issues.
  
-1. Either create your own user account as described above or use the existing "hands-on" user account. The examples will not work if run as the root account.+1. If you have problems loading the OVA image into VirtualBox, check the MD5 signature of the OVA file. The MD5 signature returned by running the program below should match the signature provided [[https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Linux-Hadoop-Minimal-0.42.MD5.txt|here]]. For each OS, use the following commands (note the name of the OVA file may be different): 
 + 
 +For **Linux** use "md5sum" 
 + 
 +  $ md5sum Linux-Hadoop-Minimal-0.42.ova 
 + 
 +For **Macintosh** use "md5" 
 + 
 +  $ md5 Linux-Hadoop-Minimal-0.42.ova 
 + 
 +For **Windows 10** (in PowerShell) use "Get-FileHash" (Also, note the use of uppercase) 
 + 
 +  C:\Users\Doug> Get-FileHash .\Linux-Hadoop-Minimal-0.42.ova -Algorithm MD5 
 + 
 +2. Either create your own user account as described above or use the existing "hands-on" user account. The examples will not work if run as the root account.
  
-2. If zip is not installed on your version of the VM, you can install it by entering the following, as root, and a "y" when asked. Zip will now be installed and available for use.+3. If zip is not installed on your version of the VM, you can install it by entering the following, as root, and a "y" when asked. Zip will now be installed and available for use.
  
    # yum install zip    # yum install zip
Line 323: Line 338:
      zip.x86_64 0:3.0-1.el6_7.1      zip.x86_64 0:3.0-1.el6_7.1
            
-3. In previous versions there is a permission issue in HDFS that prevents Hive jobs from working. To fix it, perform the following steps:+4. In previous versions there is a permission issue in HDFS that prevents Hive jobs from working. To fix it, perform the following steps:
  
 a) login to the VM as root (pw="hadoop") a) login to the VM as root (pw="hadoop")
linux_hadoop_minimal_installation_instructions.1560271414.txt.gz · Last modified: 2019/06/11 16:43 by deadline