This shows you the differences between two versions of the page.
Both sides previous revision Previous revision Next revision | Previous revision Last revision Both sides next revision | ||
linux_hadoop_minimal_installation_instructions [2019/06/11 16:10] deadline more tweaks |
linux_hadoop_minimal_installation_instructions [2020/02/13 22:03] deadline added link for LiveLessons examples |
||
---|---|---|---|
Line 1: | Line 1: | ||
- | =====Linux Hadoop Minimal Notes===== | + | =====Linux Hadoop Minimal |
- | **Version | + | **Version:** .42\\ |
- | **Date: June 3, 2019**\\ | + | **Date:** June 3, 2019\\ |
- | **Author: | + | **Author: |
- | (Email: deadline(you know what goes here)basement-supercomputing.com) | + | **Email:** deadline(you know what goes here)basement-supercomputing.com |
- | Unless otherwise noted, all course content, notes, and examples are | + | **Unless otherwise noted, all course content, notes, and examples are |
- | (c) Copyright Basement Supercomputing 2019, All rights reserved. | + | (c) Copyright Basement Supercomputing 2019, All rights reserved.** |
====What Is This?==== | ====What Is This?==== | ||
The Linux Hadoop Minimal is a virtual machine (VM) that can be used to | The Linux Hadoop Minimal is a virtual machine (VM) that can be used to | ||
- | try the examples presented in the two on-line | + | try the examples presented in the following |
* [[https:// | * [[https:// | ||
* [[https:// | * [[https:// | ||
- | It can also be used for the examples provided in the companion on-line | + | It can also be used for the [[https:// |
- | video tutorial (14+ hours) | + | video tutorial (14+ hours): |
* [[https:// | * [[https:// | ||
- | The machine has many important Hadoop and Spark packages installed and | + | The machine has many important Hadoop and Spark packages installed and at the same time tries to keep the resource usage as low as possible so the VM can used on most laptops. (See below for resource recommendations) |
- | at the same time tries to keep the resource usage as low as possible | + | |
- | so the VM can used on most laptops. (See below for resource recommendations) | + | |
To learn more about the course and my other analytics books and videos, go to: | To learn more about the course and my other analytics books and videos, go to: | ||
Line 34: | Line 32: | ||
====Student Usage==== | ====Student Usage==== | ||
- | If you have taken the " | + | If you have taken the " |
- | the '' | + | |
- | using '' | + | |
- | Zip (zip) format. It is recommended that you either make a new user account | + | |
- | or use the " | + | |
- | this account). | + | |
- | For instance, to download and extract the archive for the " | + | For instance, to download and extract the archive for the " |
wget https:// | wget https:// | ||
Line 57: | Line 50: | ||
scp -P2222 | scp -P2222 | ||
- | '' | + | '' |
- | be used for most of the examples. Therefore the command to copy file ('' | + | be used for most of the examples. Therefore, the command to copy file ('' |
- | host system to the VM is: | + | host system to the VM is (it places the file in ''/ |
scp -P2222 | scp -P2222 | ||
- | See the "Connect From Your Local Machine to the LHM Sandbox" | + | See the [[#Connect From Your Local Machine to the LHM Sandbox|Connect From Your Local Machine to the LHM Sandbox]] |
- | on using ssh and scp. | + | on using '' |
====General Usage Notes==== | ====General Usage Notes==== | ||
Line 99: | Line 92: | ||
**Step 3:** Make sure hardware virtualization is enabled in your BIOS. | **Step 3:** Make sure hardware virtualization is enabled in your BIOS. | ||
- | **Step 4:** Download | + | **Step 4:** Download the https:// |
**Step 5:** Start the VM. All the essential Hadoop service should be started automatically. | **Step 5:** Start the VM. All the essential Hadoop service should be started automatically. | ||
Line 120: | Line 114: | ||
You are should now be in the ''/ | You are should now be in the ''/ | ||
- | To confirm all the Hadoop daemons have started enter '' | + | To confirm all the Hadoop daemons have started enter '' |
- | The results should list the 10 daemons as shown below. (process numbers | + | |
- | will be different)# | + | |
< | < | ||
- | jps | + | # jps |
1938 NetworkServerControl | 1938 NetworkServerControl | ||
2036 ZeppelinServer | 2036 ZeppelinServer | ||
Line 141: | Line 133: | ||
====Copying Files In and Out of the Virtual Machine==== | ====Copying Files In and Out of the Virtual Machine==== | ||
- | To copy a file from your LOCAL MACHINE into the VM, use the '' | + | To copy a file from your LOCAL MACHINE into the VM, use the '' |
- | For instance, to copy the file '' | + | |
- | '' | + | |
- | the command places file in ''/ | + | |
scp -P2222 | scp -P2222 | ||
- | To be clear, the above command is run on your '' | + | To be clear, the above command is run on your '' |
- | On Macintosh and Linux systems run this from a terminal. On Windows | + | |
- | run it from MobaXterm. | + | |
- | To copy a file from the VM to your '' | + | To copy a file from the VM to your '' |
- | in your current directory use the following. (don't forget the '' | + | |
scp -P2222 hands-on@127.0.0.1:/ | scp -P2222 hands-on@127.0.0.1:/ | ||
Line 159: | Line 145: | ||
To be clear, the above command is run on your '' | To be clear, the above command is run on your '' | ||
- | On Windows, the data will be placed in the MobaXterm " | + | On Windows, the data will be placed in the MobaXterm " |
- | Home Directory." | + | |
- | this would be the following: | + | |
C: | C: | ||
Line 167: | Line 151: | ||
====Adding Users==== | ====Adding Users==== | ||
- | As configured, the LHM comes with one general user account. The account is called | + | As configured, the LHM comes with one general user account. The account is called |
- | Warning: Running as user '' | + | To add yourself as a user with a different user name follow the following steps. |
- | + | ||
- | To add yourself as a user. | + | |
**Step 1.** As root do the following to create a user and add a password: | **Step 1.** As root do the following to create a user and add a password: | ||
Line 305: | Line 287: | ||
and describe the situation. | and describe the situation. | ||
- | When the VM is stopped (see belwo) with '' | + | When the VM is stopped (see below) with '' |
====Stopping the VM==== | ====Stopping the VM==== | ||
Line 327: | Line 309: | ||
These issues have been addressed in the current version of the VM. Please use the lasted VM and you can avoid these issues. | These issues have been addressed in the current version of the VM. Please use the lasted VM and you can avoid these issues. | ||
- | 1. Either create your own user account as described above or use the existing " | + | 1. If you have problems loading the OVA image into VirtualBox, check the MD5 signature of the OVA file. The MD5 signature returned by running the program below should match the signature provided [[https:// |
+ | |||
+ | For **Linux** use " | ||
+ | |||
+ | $ md5sum Linux-Hadoop-Minimal-0.42.ova | ||
+ | |||
+ | For **Macintosh** use " | ||
+ | |||
+ | $ md5 Linux-Hadoop-Minimal-0.42.ova | ||
+ | |||
+ | For **Windows 10** (in PowerShell) use " | ||
+ | |||
+ | C: | ||
+ | |||
+ | 2. Either create your own user account as described above or use the existing " | ||
- | 2. If zip is not installed on your version of the VM, you can install it by entering the following, as root, and a " | + | 3. If zip is not installed on your version of the VM, you can install it by entering the following, as root, and a " |
# yum install zip | # yum install zip | ||
Line 342: | Line 338: | ||
| | ||
- | 3. In previous versions there is a permission issue in HDFS that prevents Hive jobs from working. To fix it, perform the following steps: | + | 4. In previous versions there is a permission issue in HDFS that prevents Hive jobs from working. To fix it, perform the following steps: |
- | - login to the VM as root (pw=" | + | a) login to the VM as root (pw=" |
ssh root@127.0.0.1 -p 2222 | ssh root@127.0.0.1 -p 2222 | ||
- | - then change to hdfs user | + | b) then change to hdfs user |
su - hdfs | su - hdfs | ||
- | - fix the permission error: | + | c) fix the permission error: |
hdfs dfs -chmod o+w / | hdfs dfs -chmod o+w / | ||
- | - Check the result | + | d) Check the result |
hdfs dfs -ls /user/hive | hdfs dfs -ls /user/hive | ||
- | - The output of the previous command should look like: | + | e) The output of the previous command should look like: |
Found 1 items | Found 1 items | ||
| | ||
- | - Exit out of the hdfs account | + | f) Exit out of the hdfs account |
exit | exit | ||
- | - exit out the root account | + | g) exit out the root account |
exit | exit |