Read CSV and JSON file format in spark 2.0

Read CSV with spark 2.0

STEP 1. Open the spark-shell and fire the following command.

scala> spark.read.format(“csv”).option(“header”,”true”).load(“F:/Hadoop Youtube/customer.csv”)

STEP 2. Display the result with show command

scala> .show

+—–+——+———–+——-+———-+—-+——+

|empno| ename|designation|manager| hire_date| sal|deptno|

+—–+——+———–+——-+———-+—-+——+

| 7369| SMITH| CLERK| 7902|12/17/1980| 800| 20|

| 7499| ALLEN| SALESMAN| 7698| 2/20/1981|1600| 30|

| 7521| WARD| SALESMAN| 7698| 2/22/1981|1250| 30|

| 7566|TURNER| MANAGER| 7839| 4/2/1981|2975| 20|

| 7654|MARTIN| SALESMAN| 7698| 9/28/1981|1250| 30|

| 7698|MILLER| MANAGER| 7839| 5/1/1981|2850| 30|

| 7782| CLARK| MANAGER| 7839| 6/9/1981|2450| 10|

| 7788| SCOTT| ANALYST| 7566| 12/9/1982|3000| 20|

| 7844|TURNER| SALESMAN| 7698| 9/8/1981|1500| 30|

| 7876| ADAMS| CLERK| 7788| 1/12/1983|1100| 20|

| 7900| JAMES| CLERK| 7698| 12/3/1981| 950| 30|

| 7902| FORD| ANALYST| 7566| 12/3/1981|3000| 20|

| 7934|MILLER| CLERK| 7782| 1/23/1982|1300| 10|

+—–+——+———–+——-+———-+—-+——+

scala>

Read JSON with spark 2.0

STEP 1. Read the JSON data with read method.

scala> spark.read.json(“F:/Hadoop Youtube/employee.json”)

STEP 2. display the result.

scala> .show

+—+—-+——-+

|age| id| name|

+—+—-+——-+

| 25|1201| satish|

| 28|1202|krishna|

| 39|1203| amith|

| 23|1204| javed|

| 23|1205| prudvi|

+—+—-+——-+

Leave a Reply

Your email address will not be published. Required fields are marked *