hbase - How do I make spark write to multiple sources foreachrdd -


i have made spark streaming program reads kafka source. after transformations, need send data 2 kafka producers , hbase.

i receive data via spark streaming:

customer1   21631512435 2   1449540003.803  1449540363.571  25566530    27670   1557041 19491   65664   1   197.26.8.142    197.31.74.208 customer2   21631526589 4   1449540003.821  1339540363.565  25536520    27369   1545811 19487   65659   5   197.25.2.135    197.31.74.206

i want make transformations , send 2 kafka producers , save copy hbase.

i found examples here talking sending data kafka producers , saving hbase, problem don't have sbt or maven , i'm using spark shell (spark 1.3). i've found many problems importing jars.

i'm reading kafka , saving hdfs. can me complete task?


-

Comments

Post a Comment

Popular posts from this blog

php - Permission denied. Laravel linux server -

when try register , send email user got error. working till yesterday. today gives me , cant figure out why. way on local testing work's. i'm stuck. please errorexception in compiled.php line 7512: file_put_contents(/var/www/html/storage/framework/views/53de219dee4f11a2db4f64a2c574ad02db5613f4.php): failed open stream: permission denied in compiled.php line 7512 @ handleexceptions->handleerror('2', 'file_put_contents(/var/www/html/storage/framework/views/53de219dee4f11a2db4f64a2c574ad02db5613f4.php): failed open stream: permission denied', '/var/www/html/bootstrap/cache/compiled.php', '7512', array('path' => '/var/www/html/storage/framework/views/53de219dee4f11a2db4f64a2c574ad02db5613f4.php', 'contents' => 'please activate account clicking on following link. <a href="<?php echo e(route('auth.activate', $token)); ?>"><?php echo e(route('auth.activate', $token)); ?...
Read more

google bigquery - Delta between query execution time and Java query call to finish -

context our container cluster located @ us-east1-c we using following java library: google-cloud-bigquery, 0.9.2-beta our dataset has around 26m rows , represents ~10g all of our queries return less 100 rows grouping on specific column question we analyzed last 100 queries executed in bigquery, these executed in 2-3 seconds (we analyzed calling bq --format=prettyjson show -j jobid , end time - creation time). in our java logs though, of calls bigquery.query blocking 5-6 seconds (and 10 seconds not out of ordinary). explain systematic gap between query finish in bigquery cluster , results being available in java? know 5-6 seconds isn't astronomic, curious see if normal behaviour when using java bigquery cloud library. i didn't dig point analyzed outbound call using wireshark. our tests executed in our container cluster (kubernetes). code queryrequest request = queryrequest.newbuilder(sql) .setmaxwaittime(30000l) .setuselega...
Read more

python - Pandas two dataframes multiplication? -

i have 2 data frames (a , b) a: column 1, column 2, column 3 0.1 0.5 0.7 b: row 1 5 row 2 6 row 3 7 how perform multiplication like (0.1)*5, (0.5)* 6, , (0.7)*7? in other words, how multiply value in first row of b value in first column of a, second row of b value in second column of b, , etc? you want multiply values without respect whether rows or columns. pd.series(a.values.ravel() * b.values.ravel()) 0 0.5 1 3.0 2 4.9 dtype: float64
Read more