Issue in regex_replace in Apache Spark Java -
we facing couple of issues
1. "org.apache.spark.sql.catalyst.expressions.generatedclass$generatediterator" grows beyond 64 kb".
2. "java.lang.stackoverflowerror"
first issue reported major bug in jira of apache spark https://issues.apache.org/jira/browse/spark-18492
we got these issues following program. trying replace manufacturer name equivalent alternate name,
these issues occur when have huge number of alternate names replace, small number of replacements works no issues.
datafilecontent=datafilecontent.withcolumn("manufacturersource", regexp_replace(col("manufacturersource"),str,manufacturernames.get(str).tostring()));
kindly suggest alternative method or solution go around problem.
hashtable manufacturernames = new hashtable(); enumeration names; string str; double bal; manufacturernames.put("allen","apex tool group"); manufacturernames.put("armstrong","apex tool group"); manufacturernames.put("campbell","apex tool group"); manufacturernames.put("lubriplate","apex tool group"); manufacturernames.put("delta","apex tool group"); manufacturernames.put("gearwrench","apex tool group"); manufacturernames.put("h.k. porter","apex tool group"); manufacturernames.put("jacobs","apex tool group"); manufacturernames.put("jobox","apex tool group"); manufacturernames.put("lufkin","apex tool group"); manufacturernames.put("nicholson","apex tool group"); manufacturernames.put("plumb","apex tool group"); manufacturernames.put("wiss","apex tool group"); manufacturernames.put("covert","apex tool group"); manufacturernames.put("apex-geta","apex tool group"); manufacturernames.put("dotco-airetool","apex tool group"); manufacturernames.put("apex","apex tool group"); manufacturernames.put("cleco","apex tool group"); manufacturernames.put("dotco","apex tool group"); manufacturernames.put("erem","apex tool group"); manufacturernames.put("master power","apex tool group"); manufacturernames.put("recoules quackenbush","apex tool group"); manufacturernames.put("apex-utica","apex tool group"); manufacturernames.put("weller","apex tool group"); manufacturernames.put("xcelite","apex tool group"); manufacturernames.put("jet","jpw industries"); manufacturernames.put("powermatic","jpw industries"); manufacturernames.put("wilton","jpw industries"); manufacturernames.put("black+decker","stanleyblack & decker"); manufacturernames.put("blackhawkby proto","stanleyblack & decker"); manufacturernames.put("bostitch","stanleyblack & decker"); manufacturernames.put("cribmaster","stanleyblack & decker"); manufacturernames.put("dewalt","stanleyblack & decker"); manufacturernames.put("expert (hand tools & accessories); expert (wrenches)","stanleyblack & decker"); manufacturernames.put("facom","stanleyblack & decker"); manufacturernames.put("mac","stanleyblack & decker"); manufacturernames.put("lista","stanleyblack & decker"); manufacturernames.put("porter-cable","stanleyblack & decker"); manufacturernames.put("powers","stanleyblack & decker"); manufacturernames.put("proto","stanleyblack & decker"); manufacturernames.put("stanley","stanleyblack & decker"); manufacturernames.put("vidmar","stanleyblack & decker"); manufacturernames.put("abell-howe","columbus mckinnon"); manufacturernames.put("budgit hoists","columbus mckinnon"); manufacturernames.put("cady lifters","columbus mckinnon"); manufacturernames.put("chester hoist","columbus mckinnon"); manufacturernames.put("cm","columbus mckinnon"); manufacturernames.put("coffing hoists","columbus mckinnon"); manufacturernames.put("duff norton","columbus mckinnon"); manufacturernames.put("little mule","columbus mckinnon"); manufacturernames.put("shaw-box","columbus mckinnon"); manufacturernames.put("weco","columbus mckinnon"); manufacturernames.put("yale hoist","columbus mckinnon"); manufacturernames.put("dymo","newell-rubbermaid"); manufacturernames.put("hilmor","newell-rubbermaid"); manufacturernames.put("irwin hanson","newell-rubbermaid"); manufacturernames.put("irwin marathon","newell-rubbermaid"); manufacturernames.put("irwin marples","newell-rubbermaid"); manufacturernames.put("irwin speedbor","newell-rubbermaid"); manufacturernames.put("irwin straight-line","newell-rubbermaid"); manufacturernames.put("irwin quick-grip","newell-rubbermaid"); manufacturernames.put("irwin unibit","newell-rubbermaid"); manufacturernames.put("irwin vise-grip","newell-rubbermaid"); manufacturernames.put("lenox","newell-rubbermaid"); manufacturernames.put("rubbermaid commercial products","newell-rubbermaid"); manufacturernames.put("sharpie","newell-rubbermaid"); manufacturernames.put("accu-lube","itw pro brands (itw fluids north america)"); manufacturernames.put("itw","itw pro brands (itw fluids north america)"); manufacturernames.put("rustlick","itw pro brands (itw fluids north america)"); manufacturernames.put("rustlick b","itw pro brands (itw fluids north america)"); manufacturernames.put("dymon","itw pro brands (itw fluids north america)"); manufacturernames.put("dykem","itw pro brands (itw fluids north america)"); manufacturernames.put("scrubs","itw pro brands (itw fluids north america)"); manufacturernames.put("spray nine","itw pro brands (itw fluids north america)"); manufacturernames.put("lps","itw pro brands (itw fluids north america)"); manufacturernames.put("safetap","itw pro brands (itw fluids north america)"); manufacturernames.put("sertun","itw pro brands (itw fluids north america)"); manufacturernames.put("atlantic mills","itw pro brands (itw fluids north america)"); manufacturernames.put("devcon","itw polymers adhesives"); manufacturernames.put("permatex","itw polymers adhesives"); manufacturernames.put("plexus","itw polymers adhesives"); manufacturernames.put("spraycore","itw polymers adhesives"); manufacturernames.put("norton","norton saint gobain"); manufacturernames.put("merit","norton saint gobain"); manufacturernames.put("carborundum","norton saint gobain"); manufacturernames.put("winter","norton saint gobain"); manufacturernames.put("3m","3m"); manufacturernames.put("standard abrasives","3m"); manufacturernames.put("abs","abs import tools"); manufacturernames.put("accuform","accuform mfg"); manufacturernames.put("acme","acme united"); manufacturernames.put("treds","advantage products"); manufacturernames.put("radnor","airgas"); manufacturernames.put("ajax","ajax tool works"); manufacturernames.put("akers","akers industries"); manufacturernames.put("aldon","aldon"); manufacturernames.put("gator","ali industries"); manufacturernames.put("allegro","allegro industries"); manufacturernames.put("alliance","alliance mercantile"); manufacturernames.put("viking","alliance mercantile"); manufacturernames.put("alta","alta industries"); manufacturernames.put("ammex","ammex"); manufacturernames.put("ampco","ampco safety tools"); manufacturernames.put("enforcer","amrep"); manufacturernames.put("anchor","anchor brand"); manufacturernames.put("anchor chemical","anchor chemical"); manufacturernames.put("anderson brush","anderson products"); manufacturernames.put("ansell","ansell"); manufacturernames.put("ansell","ansell"); manufacturernames.put("ansell","ansell"); manufacturernames.put("apache mills","apache mills"); manufacturernames.put("allen","apex tool group"); manufacturernames.put("allen","apex tool group"); manufacturernames.put("apex","apex tool group"); manufacturernames.put("weller","apex tool group"); manufacturernames.put("xcelite","apex tool group"); manufacturernames.put("arc","arc abrasives"); manufacturernames.put("armstrong","armstrong industrial hand tools"); manufacturernames.put("ashburn","ashburn chemical technologies"); manufacturernames.put("atlantic safety","atlantic safety products"); manufacturernames.put("baffin","baffin"); manufacturernames.put("bausch + lomb","bausch & lomb"); manufacturernames.put("bayco","bayco products"); manufacturernames.put("bayer","bayer healthcare"); manufacturernames.put("bee line","bee line abrasives"); manufacturernames.put("berne","berne apparel"); manufacturernames.put("berry","berry plastics"); manufacturernames.put("nashua","berry plastics"); manufacturernames.put("bessey","bessey tools"); manufacturernames.put("best sanitizers","best sanitizers"); manufacturernames.put("boardwalk","boardwalk"); manufacturernames.put("bdg","bob dale gloves & imports"); manufacturernames.put("bondhus","bondhus"); manufacturernames.put("boss","boss manufacturing"); manufacturernames.put("never-seez","bostik"); manufacturernames.put("bradley","bradley"); manufacturernames.put("brady","brady worldwide"); manufacturernames.put("bright star","koehler lighting products"); manufacturernames.put("bbi","brighton-best international"); manufacturernames.put("bbi","brighton-best international"); manufacturernames.put("brunner & lay","brunner & lay"); manufacturernames.put("brm","brush research mfg"); manufacturernames.put("buehler","buehler"); manufacturernames.put("buffalo","buffalo industries"); manufacturernames.put("c.h.hanson","c.h.hanson"); manufacturernames.put("c.h.hanson","c.h.hanson"); manufacturernames.put("c.s. osborne","c.s. osborne"); manufacturernames.put("cgw","camel grinding wheels"); manufacturernames.put("cgw","camel grinding wheels"); manufacturernames.put("cgw","camel grinding wheels"); manufacturernames.put("dbi-sala","capital safety"); manufacturernames.put("carborundum","carborundum abrasives"); manufacturernames.put("carborundum","carborundum abrasives"); manufacturernames.put("carhartt","carhartt"); manufacturernames.put("carolina","carolina glove"); manufacturernames.put("carr lane","carr lane mfg"); manufacturernames.put("ccl","ccl security products"); manufacturernames.put("central","central tools"); manufacturernames.put("certified safety","certified safety mfg"); manufacturernames.put("channellock","channellock"); manufacturernames.put("chase","chase ergonomics"); manufacturernames.put("checkers","checkers industrial safety products"); manufacturernames.put("cp","chicago pneumatic"); manufacturernames.put("cp","chicago pneumatic"); manufacturernames.put("cpa","chicago protective apparel"); manufacturernames.put("choctaw-kaul","choctaw - kaul distribution"); manufacturernames.put("climax","climax metal products"); manufacturernames.put("cmc rescue","cmc rescue"); manufacturernames.put("coastal wipers","coastal wipers"); manufacturernames.put("codet","codet"); manufacturernames.put("comfort","comfort clothing & gloves"); manufacturernames.put("comfort eye protection","comfort eye protection"); manufacturernames.put("cep","complete enviromental products"); manufacturernames.put("condor","condor protective gear"); manufacturernames.put("cordova","cordova safety products"); manufacturernames.put("cortina","cortina"); manufacturernames.put("cox","cox north america"); manufacturernames.put("craftsman","craftsman"); manufacturernames.put("craftsman","craftsman"); manufacturernames.put("craftsman","craftsman"); manufacturernames.put("cratex","cratex mfg"); manufacturernames.put("crc","crc industries"); manufacturernames.put("crossfire","crossfire safety eyewear"); manufacturernames.put("crown-mats","crown matting technologies"); manufacturernames.put("cs unitec","cs unitec"); manufacturernames.put("clc","custom leather craft"); manufacturernames.put("dap","dap products"); manufacturernames.put("dascopro","dasco pro tools"); manufacturernames.put("dascopro","dasco pro tools"); manufacturernames.put("degil","degil safety products"); manufacturernames.put("dentec","dentec safety specialists"); manufacturernames.put("dorcy","dorcy international"); manufacturernames.put("dormer pramet","dormer pramet"); manufacturernames.put("precision dormer","dormer pramet"); manufacturernames.put("dräger","draeger safety"); manufacturernames.put("dupont","dupont protective apparel"); manufacturernames.put("duracell","duracell"); manufacturernames.put("dynabrade","dynabrade"); manufacturernames.put("eagle","eagle manufacturing"); manufacturernames.put("bussmann","eaton"); manufacturernames.put("bullard","ed bullard"); manufacturernames.put("eklind","eklind tool company"); manufacturernames.put("eklind","eklind tool company"); manufacturernames.put("elk river","elk river"); manufacturernames.put("elmers","elmers products"); manufacturernames.put("elvex","elvex"); manufacturernames.put("ridgid","emerson electric"); manufacturernames.put("empire","empire level"); manufacturernames.put("emuge","emuge"); manufacturernames.put("encon","encon safety products"); manufacturernames.put("energizer","energizer holdings"); manufacturernames.put("erb","erb industries"); manufacturernames.put("ergodyne","ergodyne"); manufacturernames.put("esca","esca tech"); manufacturernames.put("esp","esp"); manufacturernames.put("estwing","estwing mfg"); manufacturernames.put("estwing","estwing mfg"); manufacturernames.put("everett","everett industries"); manufacturernames.put("extech","extech instruments"); manufacturernames.put("falcon abrasive","falcon abrasive"); manufacturernames.put("falcon abrasive","falcon abrasive"); manufacturernames.put("falcon abrasive","falcon abrasive"); manufacturernames.put("falltech","falltech"); manufacturernames.put("fibre-metal honeywell","honeywell safety products"); manufacturernames.put("first aid only","first aid only"); manufacturernames.put("flexovit","flexovit"); manufacturernames.put("flexovit","flexovit"); manufacturernames.put("fluke","fluke"); manufacturernames.put("forcefield","forcefield"); manufacturernames.put("titebond","franklin international"); manufacturernames.put("franmar","franmar"); manufacturernames.put("fuller","fuller"); manufacturernames.put("gants laurentide","gants laurentide"); manufacturernames.put("gardner-gibson","gardner-gibson"); manufacturernames.put("garland","garland mfg"); manufacturernames.put("garr tool","garr tool"); manufacturernames.put("gates","gates"); manufacturernames.put("gateway safety","gateway safety"); manufacturernames.put("gatorade","gatorade"); manufacturernames.put("gedore","gedore tools"); manufacturernames.put("general","general tools & instruments"); manufacturernames.put("generic","generic"); manufacturernames.put("gf","gf protection"); manufacturernames.put("global glove","global glove & safety mfg"); manufacturernames.put("global glove","global glove & safety mfg"); manufacturernames.put("gorilla","gorilla glue"); manufacturernames.put("lutz","gorilla glue"); manufacturernames.put("greatneck","greatneck tools"); manufacturernames.put("greatneck","greatneck tools"); manufacturernames.put("greenfield industries","greenfield industries"); manufacturernames.put("greenlee","greenlee textron"); manufacturernames.put("greenlee","greenlee textron"); manufacturernames.put("gp","grey pneumatic"); manufacturernames.put("grobet","grobet"); manufacturernames.put("guardian","guardian equipment"); manufacturernames.put("guard-line","guard-line"); manufacturernames.put("guhring","guhring"); manufacturernames.put("harley-davidson","harley-davidson safety wear"); manufacturernames.put("harris","harris industries"); manufacturernames.put("hart health","hart health"); manufacturernames.put("haws","haws"); manufacturernames.put("helly-hansen","helly-hansen"); manufacturernames.put("henkel","henkel"); manufacturernames.put("hermes","hermes abrasives"); manufacturernames.put("hexarmor","hex armor"); manufacturernames.put("honeywell","honeywell safety products"); manufacturernames.put("hyde","hyde tools"); manufacturernames.put("hygiena","hygiena"); manufacturernames.put("hygrade","hygrade/ldf industries"); manufacturernames.put("hy-tech","hy-tech machine"); manufacturernames.put("hytest","hytest safety footwear"); manufacturernames.put("ideal","ideal industries"); manufacturernames.put("honest abe","illinois glove"); manufacturernames.put("itw","itw global brands"); manufacturernames.put("impacto","impacto protective products"); manufacturernames.put("impacto","impacto protective products"); manufacturernames.put("enviroguard","international enviroguard systems"); manufacturernames.put("ipg","intertape polymer group"); manufacturernames.put("ironclad","ironclad performance wear"); manufacturernames.put("ironclad","ironclad performance wear"); manufacturernames.put("ironclad","ironclad performance wear"); manufacturernames.put("ironwear","ironwear rainwear"); manufacturernames.put("irwin","irwin tools"); manufacturernames.put("irwin","irwin tools"); manufacturernames.put("iscar","iscar"); manufacturernames.put("devcon","itw devcon"); manufacturernames.put("itw","itw pro brands"); manufacturernames.put("itw","itw pro brands"); manufacturernames.put("j&m","j&m innovative products"); manufacturernames.put("jbc","jbc safety plastic"); manufacturernames.put("jet lube","jet lube"); manufacturernames.put("jisco","jisco abrasives"); manufacturernames.put("tillman","john tillman"); manufacturernames.put("johnson & johnson","johnson & johnson services"); manufacturernames.put("jomac","wells lamont industrial"); manufacturernames.put("jpw","jpw industries"); manufacturernames.put("steelman","js products"); manufacturernames.put("justrite","justrite mfg"); manufacturernames.put("justrite","justrite mfg"); manufacturernames.put("kaba","kaba ilco"); manufacturernames.put("kasco","kasco abrasives"); manufacturernames.put("hotmax","kdar"); manufacturernames.put("keystone","keystone adjustable cap"); manufacturernames.put("kimberly clark*","kimberly-clark professional"); manufacturernames.put("kimberly clark*","kimberly-clark professional"); manufacturernames.put("kimberly clark*","kimberly-clark professional"); manufacturernames.put("kinco","kinco international"); manufacturernames.put("kinco","kinco international"); manufacturernames.put("kirchhoff","kirchhoff group"); manufacturernames.put("witte","kirchhoff group"); manufacturernames.put("klein","klein tools"); manufacturernames.put("klingspor","klingspor abrasives"); manufacturernames.put("komelon","komelon"); manufacturernames.put("kool mist","kool mist"); manufacturernames.put("krylon","krylon products group"); manufacturernames.put("krylon","krylon products group"); manufacturernames.put("mirka","kwh mirka"); manufacturernames.put("starrett","l.s. starrett"); manufacturernames.put("la-co","la-co industries"); manufacturernames.put("markal","la-co industries"); manufacturernames.put("lacrosse","lacrosse footwear"); manufacturernames.put("lakeland","lakeland industries"); manufacturernames.put("a & e","lang tools"); manufacturernames.put("lapco fr","lapco mfg"); manufacturernames.put("leatherman","leatherman tool group"); manufacturernames.put("lenox","lenox"); manufacturernames.put("lenox","lenox"); manufacturernames.put("lfs","lfs glove & safety"); manufacturernames.put("liberty glove","liberty glove"); manufacturernames.put("lincoln electric","lincoln electric"); manufacturernames.put("lixie","lixie"); manufacturernames.put("lp royer","lp royer"); manufacturernames.put("lubriplate","lubriplate lubricants"); manufacturernames.put("lyndex-nikken","lyndex-nikken"); manufacturernames.put("m.a. ford","m.a. ford"); manufacturernames.put("maglite","mag instrument"); manufacturernames.put("magid","magid glove"); manufacturernames.put("majestic glove","majestic glove"); manufacturernames.put("makita","makita"); manufacturernames.put("makita","makita"); manufacturernames.put("malt","malt industries"); manufacturernames.put("mapa","mapa professional"); manufacturernames.put("martin","martin sprocket & gear"); manufacturernames.put("martin","martin sprocket & gear"); manufacturernames.put("master chemical","master chemical corp"); manufacturernames.put("master lock","master lock"); manufacturernames.put("maxcraft","maxcraft"); manufacturernames.put("mayhew","mayhew steel products"); manufacturernames.put("mayhew","mayhew steel products"); manufacturernames.put("workhorse","mccordick glove & safety"); manufacturernames.put("mcr safety","mcr safety"); manufacturernames.put("river city","mcr safety"); manufacturernames.put("u.s. safety","mcr safety"); manufacturernames.put("mechanix wear","mechanix wear"); manufacturernames.put("medique","medique products"); manufacturernames.put("merit","norton saint gobain"); manufacturernames.put("merit","norton saint gobain"); manufacturernames.put("merit","norton saint gobain"); manufacturernames.put("metabo","metabo"); manufacturernames.put("microflex","ansell"); manufacturernames.put("milwaukee","milwaukee tool"); manufacturernames.put("ml kishigo","ml kishigo"); manufacturernames.put("moldex","moldex"); manufacturernames.put("moldex","moldex"); manufacturernames.put("morse","morse cutting tools"); manufacturernames.put("mro solutions","mro solutions"); manufacturernames.put("msa","msa safety"); manufacturernames.put("nasco","nasco industries"); manufacturernames.put("nmc","national marker"); manufacturernames.put("nsa","national safety apparel"); manufacturernames.put("neese","neese industries"); manufacturernames.put("pig","new pig"); manufacturernames.put("newborn","newborn brothers"); manufacturernames.put("newell-rubbermaid","newell-rubbermaid"); manufacturernames.put("ngk berylco","ngk metals"); manufacturernames.put("nite ize","nite ize"); manufacturernames.put("nitto kohki","nitto kohki"); manufacturernames.put("north honeywell","honeywell safety products"); manufacturernames.put("north star glove","north star glove"); manufacturernames.put("norton","norton saint gobain"); manufacturernames.put("norton","norton saint gobain"); manufacturernames.put("norton","norton saint gobain"); manufacturernames.put("oateyscs","oateyscs"); manufacturernames.put("oberon","oberon"); manufacturernames.put("miracool","occunomix international"); manufacturernames.put("olympia","olympia tools"); manufacturernames.put("omar","omar"); manufacturernames.put("onguard","ansell"); manufacturernames.put("orr safety","orr safety"); manufacturernames.put("osborn","osborn"); manufacturernames.put("osg","osg"); manufacturernames.put("osg","osg"); manufacturernames.put("osg","osg"); manufacturernames.put("ossian","ossian"); manufacturernames.put("pac-kit","acme united"); manufacturernames.put("panduit","panduit"); manufacturernames.put("paulson","paulson mfg"); manufacturernames.put("pearl","pearl abrasive"); manufacturernames.put("pelican","pelican products"); manufacturernames.put("petzl","petzl"); manufacturernames.put("pferd","pferd"); manufacturernames.put("pony","pony tools"); manufacturernames.put("pony","pony tools"); manufacturernames.put("pony","pony tools"); manufacturernames.put("pratt-read","pratt-read tools"); manufacturernames.put("precision abrasives","precision abrasives"); manufacturernames.put("pta","premium tool & abrasives"); manufacturernames.put("presco","presco"); manufacturernames.put("caiman","primax mfg & trading"); manufacturernames.put("primax","primax mfg & trading"); manufacturernames.put("bouton","protective industrial products"); manufacturernames.put("pip","protective industrial products"); manufacturernames.put("pyramex","pyramex safety products"); manufacturernames.put("qrp","qrp gloves & fingercots"); manufacturernames.put("quality import","quality import"); manufacturernames.put("quest","quest environmental & safety products"); manufacturernames.put("radiac","radiac abrasives"); manufacturernames.put("radians","radians"); manufacturernames.put("rsc","radiator specialty"); manufacturernames.put("rawhyde frontier","rawhyde frontier"); manufacturernames.put("rectorseal","rectorseal"); manufacturernames.put("red devil","red devil"); manufacturernames.put("red kap","red kap industries"); manufacturernames.put("reed","reed mfg"); manufacturernames.put("reed","reed mfg"); manufacturernames.put("revco","revco"); manufacturernames.put("rex-cut","rex-cut abrasives"); manufacturernames.put("ridgid","ridgid tool"); manufacturernames.put("ridgid","ridgid tool"); manufacturernames.put("ridgid","ridgid tool"); manufacturernames.put("ridgid","ridgid tool"); manufacturernames.put("ri","rigid industries"); manufacturernames.put("ringers","ringers gloves"); manufacturernames.put("dremel","robert bosch"); manufacturernames.put("bosch","robert bosch"); manufacturernames.put("bosch","robert bosch"); manufacturernames.put("round house","round house"); manufacturernames.put("rpb","rpb safety"); manufacturernames.put("rust-oleum","rust-oleum"); manufacturernames.put("rutland","rutland fire clay"); manufacturernames.put("safco","safco products"); manufacturernames.put("safecross","safecross (first aid products)"); manufacturernames.put("saf-tech","saf-tech"); manufacturernames.put("saf-t-glove","saf-t-glove"); manufacturernames.put("salisbury honeywell","honeywell safety products"); manufacturernames.put("san jamar","san jamar"); manufacturernames.put("sas","sas safety"); manufacturernames.put("sashco","sashco"); manufacturernames.put("sf","seal fast"); manufacturernames.put("sg","seattle glove"); manufacturernames.put("sellstrom","sellstrom"); manufacturernames.put("sellstrom","sellstrom"); manufacturernames.put("sempermed","sempermed"); manufacturernames.put("serim","serim research"); manufacturernames.put("seymour midwest","seymour midwest"); manufacturernames.put("seymour","seymour of sycamore"); manufacturernames.put("sgs","boss manufacturing"); manufacturernames.put("showa best","showa best glove"); manufacturernames.put("showa best","showa best glove"); manufacturernames.put("shurtape","shurtape technologies"); manufacturernames.put("shurtape","shurtape technologies"); manufacturernames.put("sia","sia abrasives industries"); manufacturernames.put("simonds","simonds international"); manufacturernames.put("sk","sk hand tool"); manufacturernames.put("bahco","snap-on industrial"); manufacturernames.put("williams","snap-on industrial"); manufacturernames.put("snap-on","snap-on industrial"); manufacturernames.put("snap-on","snap-on industrial"); manufacturernames.put("snap-on","snap-on industrial"); manufacturernames.put("snap-on","snap-on industrial"); manufacturernames.put("williams","snap-on industrial"); manufacturernames.put("soudal accumetric","soudal accumetric"); manufacturernames.put("speakman","speakman"); manufacturernames.put("rayovac","spectrum brands"); manufacturernames.put("spilltech","spilltech environmental"); manufacturernames.put("stabilicer","stabilicer"); manufacturernames.put("stanco","stanco mfg"); manufacturernames.put("stanco","stanco mfg"); manufacturernames.put("standard safety","standard safety equipment company"); manufacturernames.put("standard safety","standard safety equipment company"); // show balances in hash table. names = manufacturernames.keys(); dataset<row> datafilecontent = sqlcontext.load("com.databricks.spark.csv", options); while(names.hasmoreelements()) { str = (string) names.nextelement(); datafilecontent=datafilecontent.withcolumn("manufacturersource", regexp_replace(col("manufacturersource"),str,manufacturernames.get(str).tostring())); } datafilecontent.show();
with big count of regexp_replace operations have forced stackoverflow. have 3 options:
- dirty: enlarge stack size drivers , executors (google java options stack size)
- write 1 udf function regexp_replace in 1 column edit operation (with tools of programming language). big count of regexp replaces unefficient. can try aho corrasick algorithm
- switch operation join. make rdd/dataframe replace table sc.parallelize join name , coalesce(newname, oldname)
the more replaces have more should go 3rd way. guess if have @ least thousands of replaces far better way
Comments
Post a Comment