we have ambari cluster with 3 masters machines , two kafka’s and 3 workers machines each worker have 5 disks as : we want to add additional 5 disks to each worker machine as ( /dev/sdf , /dev/sdh …, etc ) remark – our Workers nodes runs both a DataNode and NodeManager for now I understand that the following parameters
Tag: hdfs
Error while decompress a file from Local Linux to HDFS
This command works fine in Local linux I would like to decompress a file with extension .csv.gz to HDFS location. I tried the below command and i get this error Could someone help me to fix this? Answer To make gzip write the output on standard output, use -c flag. So the command would be,
How to store log files of a shell script in HDFS
I have shell script in HDFS. I want to collect the logs for this script in HDFS only. The contents of the script are below: The logs are not appending to the files. only the files are being created. How can I get the files to have the result of the function to be appended in HDFS Answer The logs
Setting up Hadoop in Pseudo-distributed mode in ubuntu
I’m trying to teach myself Hadoop on my laptop. My objective is to get the pseudo distributed mode running. I’m following the guide from the Apache website to set up Hadoop and HDFS in Ubuntu, but I can’t get it to work. Here are the steps I have followed so far: 1) check Java version: returns: 2) obtain Hadoop 2.7:
Cannot write to Hadoop DFS directory mode 775 group permission UserGroupInformation
I’m running Hadoop 2.6.2 on a private cluster with file-system permissions enabled. The cluster has password files with only system users like hadoop, no personal accounts. I’m accessing DFS from a linux edge node that has personal accounts like mine (‘clott’). The problem is that I cannot write to a DFS directory (‘shared’) that is mode 775 and group hadoop;