I am trying to load over 300,000,000 millions rows of data into a mysql table and I am trying to make it faster. However, when I move the prepare statement out of the loop and place the $uwind and $vwind in the execute it won't work. It says the data for $uwind is truncated. Can someone help me out here? Or, if you know of a faster way to load 300,000,000 millions rows of data from text files into mysql that would be greatly appreciated! Thanks a lot. Code: $filename = "1980\\1\\" . $filename; #Open's the file with the data. First will be the U wind data open(FILE, $filename) or die "$!"; @uwind = <FILE>; $filename =~ s/u.txt/v.txt/; open(FILE, $filename) or die "$!"; @vwind = <FILE>; $sth=$dbh->prepare ("INSERT INTO 10m (ID, DATE, TIME, LATITUDE, LONGITUDE, X, Y, U, V, RESULTANT, VARIANCE_RESULTANT, ANGLE, VARIANCE_ANGLE) VALUES ($id, '$date', '$hour', ?, ?, ?, ?, $uwind[$i], $vwind[$i], ?, ?, ?, ?)"); for ($y=-4612.566; $y<=4300; $y=$y+32.463) { for ($x=-5632.668; $x<=5600; $x=$x+32.463) { $id= $id + 1; $i = $i + 1; if ($uwind[$i] < 100 && $vwind[$i] < 100) { $rho = sqrt($x*$x + ($rhozero - $y)*($rhozero - $y)); $theta = atan2($x,($rhozero - $y)); $lat = (2*atan(($R*$F/$rho)**(1/$N)) - $pi/2) * $degs; $longi = (($theta/$N) * $degs) + $lambda0; $resultant = sqrt($uwind[$i]**2 + $vwind[$i]**2); $angle = (atan2($vwind[$i],$uwind[$i])) * $degs; if ($angle<0) { $angle=$angle + 360; } $variance_speed = 0; $variance_angle = 0; $sth->execute ($lat, $longi, $x, $y, $resultant, $variance_speed, $angle, $variance_angle) || die "Couldn't insert record : $DBI::errstr";
I just attended this seminar yesterday and today I'm recommending it. COOL Try this one, Oracle Data Integrator. This just suites your requirement. 1] Load data from Text files to MySQL 2] Load tons of data in min amount of time The presenter had loaded data from a 30GB file in around 30 minutes. http://www.oracle.com/technology/software/products/odi/index.html