스파크 설치
DB CAFE
thumb_up 추천메뉴 바로가기
- DBA { Oracle DBA 명령어 > DBA 초급 과정 > DBA 고급 과정 }
- 튜닝 { 오라클 튜닝 목록 }
- 모델링 { 데이터 모델링 가이드 }
목차
1 스파크 설치[편집]
1.1 설치에 필요한 파일 다운로드[편집]
- 스파크_하둡 https://spark.apache.org/downloads.html
- JAVA JDK
- WINUTILS https://github.com/cdarlint/winutils
1.2 PySpark 설치 (pip 이용)[편집]
pip install pysparkSet the necessary environment variables. PySpark needs to know the path to your Java installation.
1.3 Linux or macOS 에서 설치[편집]
export JAVA_HOME=<path_to_java>
export SPARK_HOME=$(python -c "import site; print(site.getsitepackages()[0])")/pyspark
export PYSPARK_PYTHON=python3
export PYSPARK_DRIVER_PYTHON=python3Replace <path_to_java> with the actual path to your Java installation. You can find it using the following command:
/usr/libexec/java_home1.4 Windows 에서 설치[편집]
$env:JAVA_HOME = "<path_to_java>"
$env:SPARK_HOME = (python -c "import site; print(site.getsitepackages()[0])") + "\pyspark"
$env:PYSPARK_PYTHON = "python3"
$env:PYSPARK_DRIVER_PYTHON = "python3"Replace <path_to_java> with the actual path to your Java installation.
1.5 PySpark 설치 후 테스트[편집]
Create a new Python script or open an interactive Python shell, and run the following code:
from pyspark.sql import SparkSession
spark = SparkSession.builder \
.appName("PySpark Test") \
.getOrCreate()
data = [("Alice", 1), ("Bob", 2), ("Cathy", 3)]
columns = ["Name", "ID"]
df = spark.createDataFrame(data, columns)
df.show()If the installation is successful, you will see the DataFrame displayed as output.
(Optional) Install additional packages: You may need additional packages for your PySpark projects, such as pandas, numpy, or matplotlib. Install them using pip:
pip install pandas numpy matplotlibWith PySpark installed, you can now start working on your data processing and machine learning projects.