Go4Expert

Go4Expert (http://www.go4expert.com/)
-   Perl (http://www.go4expert.com/forums/perl/)
-   -   Why won't string work?? (http://www.go4expert.com/forums/wont-string-t18803/)

wecc 4Aug2009 06:45

Why won't string work??
 
I am trying to load over 300,000,000 millions rows of data into a mysql table and I am trying to make it faster. However, when I move the prepare statement out of the loop and place the $uwind and $vwind in the execute it won't work. It says the data for $uwind is truncated. Can someone help me out here? Or, if you know of a faster way to load 300,000,000 millions rows of data from text files into mysql that would be greatly appreciated! Thanks a lot.

Code:


$filename = "1980\\1\\" . $filename;

                #Open's the file with the data. First will be the U wind data

                open(FILE, $filename) or die "$!";
                @uwind = <FILE>;

                $filename =~ s/u.txt/v.txt/;

                open(FILE, $filename) or die "$!";
                @vwind = <FILE>;
                $sth=$dbh->prepare
                                ("INSERT INTO 10m    (ID, DATE, TIME, LATITUDE, LONGITUDE, X, Y, U, V, RESULTANT, VARIANCE_RESULTANT, ANGLE, VARIANCE_ANGLE)
                                VALUES            ($id, '$date', '$hour', ?, ?, ?, ?, $uwind[$i], $vwind[$i], ?, ?, ?, ?)");
                for ($y=-4612.566; $y<=4300; $y=$y+32.463) {
            for ($x=-5632.668; $x<=5600; $x=$x+32.463) {

                                $id= $id + 1;
                                $i = $i + 1;



                              if ($uwind[$i] < 100 && $vwind[$i] < 100) {
                             
                                $rho = sqrt($x*$x + ($rhozero - $y)*($rhozero - $y));
                                $theta = atan2($x,($rhozero - $y));
                                $lat = (2*atan(($R*$F/$rho)**(1/$N)) - $pi/2) * $degs;
                                $longi = (($theta/$N) * $degs) + $lambda0;

                                $resultant = sqrt($uwind[$i]**2 + $vwind[$i]**2);
                                $angle = (atan2($vwind[$i],$uwind[$i])) * $degs;
                                        if ($angle<0) {
                                        $angle=$angle + 360;
                                      }

                                $variance_speed = 0;
                                $variance_angle = 0;

                                $sth->execute ($lat, $longi, $x, $y, $resultant, $variance_speed, $angle, $variance_angle) || die "Couldn't insert record : $DBI::errstr";


nimesh 5Aug2009 23:41

Re: Why won't string work??
 
I just attended this seminar yesterday and today I'm recommending it. COOL :)

Try this one, Oracle Data Integrator.
This just suites your requirement.

1] Load data from Text files to MySQL
2] Load tons of data in min amount of time

The presenter had loaded data from a 30GB file in around 30 minutes.

http://www.oracle.com/technology/sof...odi/index.html


All times are GMT +5.5. The time now is 15:16.