Both sides previous revision
Previous revision
Next revision
|
Previous revision
|
linux_hadoop_minimal_installation_instructions_version_2 [2024/01/05 21:37] deadline [Linux Hadoop Minimal VM Notes VERSION 2] |
linux_hadoop_minimal_installation_instructions_version_2 [2024/01/30 14:07] (current) deadline [Issues/Bugs] |
| |
=== Intel x86_64 with VirtualBox=== | === Intel x86_64 with VirtualBox=== |
* Version: 2.0-beta8\\ | * Version: 2.0-8.1\\ |
* Release Date: August 9, 2022 | * Release Date: 25-Jan-2024 |
| |
=== Apple M1, M2 with UTM === | === Apple M1, M2 with UTM === |
* Version: 2.0-beta8-M1-R2 | * Version: 2.0-M8.1 |
* Release Date: February 20, 2023 | * Release Date: 25-Jan-2024 |
| |
=== Issues with Either version of the LHM=== | === Issues with Either version of the LHM=== |
**Step 3a:** On MacIntosh Systems with //Big Sur//, you may get the ''Kernel driver not installed (rc=-1908)'' error. The error is do to new security levels in MacOS. See this [[ https://www.howtogeek.com/658047/how-to-fix-virtualboxs-%E2%80%9Ckernel-driver-not-installed-rc-1908-error/|page]] for a fix. | **Step 3a:** On MacIntosh Systems with //Big Sur//, you may get the ''Kernel driver not installed (rc=-1908)'' error. The error is do to new security levels in MacOS. See this [[ https://www.howtogeek.com/658047/how-to-fix-virtualboxs-%E2%80%9Ckernel-driver-not-installed-rc-1908-error/|page]] for a fix. |
| |
**Step 4:** Download the current Linux-Hadoop-Minimal-V2.0-xxxxx.ova image [[http://161.35.229.207/download/Linux-Hadoop-Minimal-V2.0-beta8.ova|US]] [[http://134.209.239.225/download/Linux-Hadoop-Minimal-V2.0-beta8.ova|Europe]] and load it into VirtualBox. | **Step 4:** Download the current Linux-Hadoop-Minimal-V2.0-xxxxx.ova image [[http://161.35.229.207/download/Linux-Hadoop-Minimal-V2.0-8.1.ova|US]] [[http://134.209.239.225/download/Linux-Hadoop-Minimal-V2.0-8.1.ova|Europe]] and load it into VirtualBox. |
| |
**Step 5:** Start the VM. All the essential Hadoop service should be started automatically. | **Step 5:** Start the VM. All the essential Hadoop service should be started automatically. |
**IMPORTANT** The LHM should be stopped when not in use. The running services should be stopped in a ''graceful manner'' (powered down) as frequent standby or a sudden power interruption can leave some services in a broken state. To stop the LHM, it is suggested the you login as the root user (see [[linux_hadoop_minimal_installation_instructions_version_2#stopping_the_vm|Stopping the VM]] ) and issue the ''poweroff'' command. This will ensure a safe and orderly shutdown of the machine. | **IMPORTANT** The LHM should be stopped when not in use. The running services should be stopped in a ''graceful manner'' (powered down) as frequent standby or a sudden power interruption can leave some services in a broken state. To stop the LHM, it is suggested the you login as the root user (see [[linux_hadoop_minimal_installation_instructions_version_2#stopping_the_vm|Stopping the VM]] ) and issue the ''poweroff'' command. This will ensure a safe and orderly shutdown of the machine. |
| |
====Installation Steps for Apple-M1 Based Hosts ==== | ====Installation Steps for Apple-M Based Hosts ==== |
| |
**20-Feb-2023 VERSION-UPDATED** | **25-Jan-2024 VERSION-UPDATED** |
| |
There is a partial LHM for the Apple M1 based machines. The following (aarch64) packages are installed: CentOS Linux 7.6, Python3 3.6.8, R 3.6.0, Hadoop 3.3.0, Hive 3.1.2, Apache Spark 2.4.5, Derby 10.14.2.0, Zeppelin 0.8.2, Sqoop 1.4.7, Kafka 2.5.0, HBase 2.4.10, NiFi 1.17.0 (KafkaEsque is not installed) | There is a full LHM for the Apple M based machines. The following (aarch64) packages are installed: CentOS Linux 7.6, Python3 3.6.8, R 3.6.0, Hadoop 3.3.0, Hive 3.1.2, Apache Spark 2.4.5, Derby 10.14.2.0, Zeppelin 0.8.2, Sqoop 1.4.7, Kafka 2.5.0, HBase 2.4.10, NiFi 1.17.0 (KafkaEsque is not installed) |
| |
To run the VM on an Apple-M1 based machine, Perform the following steps: | To run the VM on an Apple-M1 based machine, Perform the following steps: |
| |
| |
**Step 2:** Download the LHM for Apple-M1 into your ''Downloads'' folder. The current version is [[http://161.35.229.207/download/Linux-Hadoop-Minimal-V2.0-beta8-M1-R2.utm.zip|US]] [[http://134.209.239.225/download/Linux-Hadoop-Minimal-V2.0-beta8-M1-R2.utm.zip|Europe]] (11G) The [[https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Linux-Hadoop-Minimal-V2.0-beta8-M1-R2.MD5.txt| MD5 file ]] can be used to verify the integrity of the download. **NOTE:** Google Chrome may prevent http downloads, right click the link, choose “Save Link As” then click “Keep” next to the blue discard box at the bottom of the browser. | **Step 2:** Download the LHM for Apple-M1 into your ''Downloads'' folder. The current version is [[http://161.35.229.207/download/Linux-Hadoop-Minimal-V2.0-M8.1.utm.zip|US]] [[http://134.209.239.225/download/Linux-Hadoop-Minimal-V2.0-M8.1.utm.zip|Europe]] (9G) The [[https://www.clustermonkey.net/download/Hands-on_Hadoop_Spark/Linux-Hadoop-Minimal-V2.0-M8.1.utm.zip.MD5.txt| MD5 file ]] can be used to verify the integrity of the download. **NOTE:** Google Chrome may prevent http downloads, right click the link, choose “Save Link As” then click “Keep” next to the blue discard box at the bottom of the browser. |
| |
**Step 3:** The downloaded LHM is a zip file. Click to extract the file. It should create a directory called ''Linux-Hadoop-Minimal-V2.0-beta8-M1-R2.utm'' | **Step 3:** The downloaded LHM is a zip file. Click to extract the file. It should create a directory called ''Linux-Hadoop-Minimal-V2.0-M8.1.utm'' |
| |
**Step 4:** Start UTM (it will be in the applications folder, suggest keeping it in your dock) Click the ''+'' to the left of the ''UTM'' at the top. A window called ''Start'' will open. Toward the bottom click ''Open'' under ''Existing''. Next, navigate to the ''Downloads'' folder and click on ''Linux-Hadoop-Minimal-V2.0-beta8-M1-R2.utm'' The LHM should now be listed in the left hand column in the UTM window. | **Step 4:** Start UTM (it will be in the applications folder, suggest keeping it in your dock) Click the ''+'' to the left of the ''UTM'' at the top. A window called ''Start'' will open. Toward the bottom click ''Open'' under ''Existing''. Next, navigate to the ''Downloads'' folder and click on ''Linux-Hadoop-Minimal-V2.0-M8.1.utm'' The LHM should now be listed in the left hand column in the UTM window. |
| |
**Step 5:** Start the LHM by clicking on the big arrow in the middle of the window. (If it is not highlighted, click on the ''Linux-Hadoop-Minimal-V2.0-beta8-M1-R2'' on the left) The LHM should start. A new "terminal" window will open and show the VM boot-up sequence. When finished, a prompt will be displayed in this window (''localhost login:'') Minimize this terminal window. We will log into the LHM using the Mac terminal program. | **Step 5:** Start the LHM by clicking on the big arrow in the middle of the window. (If it is not highlighted, click on the ''Linux-Hadoop-Minimal-V2.0-beta8-M1-R2'' on the left) The LHM should start. A new "terminal" window will open and show the VM boot-up sequence. When finished, a prompt will be displayed in this window (''localhost login:'') Minimize this terminal window. We will log into the LHM using the Mac terminal program. |
3. If the **time on the LHM falls out of sync** with the host due to hibernation, the following commands can be run to reset the the ntpd time daemon (run as root). **NOTE:** the host must have Internet access. | 3. If the **time on the LHM falls out of sync** with the host due to hibernation, the following commands can be run to reset the the ntpd time daemon (run as root). **NOTE:** the host must have Internet access. |
| |
systemctl -l start ntpd | systemctl -l stop ntpd |
ntpdate -u pool.ntp.org | ntpdate -u pool.ntp.org |
systemctl -l start ntpd | systemctl -l start ntpd |