1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Why won't string work??

Discussion in 'Perl' started by wecc, Aug 4, 2009.

  1. wecc

    wecc New Member

    Joined:
    Aug 4, 2009
    Messages:
    2
    Likes Received:
    0
    Trophy Points:
    0
    I am trying to load over 300,000,000 millions rows of data into a mysql table and I am trying to make it faster. However, when I move the prepare statement out of the loop and place the $uwind and $vwind in the execute it won't work. It says the data for $uwind is truncated. Can someone help me out here? Or, if you know of a faster way to load 300,000,000 millions rows of data from text files into mysql that would be greatly appreciated! Thanks a lot.

    Code:
    
    $filename = "1980\\1\\" . $filename;
    
                    #Open's the file with the data. First will be the U wind data
    
                    open(FILE, $filename) or die "$!";
                    @uwind = <FILE>;
    
                    $filename =~ s/u.txt/v.txt/;
    
                    open(FILE, $filename) or die "$!";
                    @vwind = <FILE>;
                    $sth=$dbh->prepare
                                    ("INSERT INTO 10m     (ID, DATE, TIME, LATITUDE, LONGITUDE, X, Y, U, V, RESULTANT, VARIANCE_RESULTANT, ANGLE, VARIANCE_ANGLE)
                                    VALUES            ($id, '$date', '$hour', ?, ?, ?, ?, $uwind[$i], $vwind[$i], ?, ?, ?, ?)");
                    for ($y=-4612.566; $y<=4300; $y=$y+32.463) {
                for ($x=-5632.668; $x<=5600; $x=$x+32.463) {
    
                                    $id= $id + 1;
                                    $i = $i + 1;
    
    
    
                                   if ($uwind[$i] < 100 && $vwind[$i] < 100) {
                                   
                                    $rho = sqrt($x*$x + ($rhozero - $y)*($rhozero - $y));
                                    $theta = atan2($x,($rhozero - $y));
                                    $lat = (2*atan(($R*$F/$rho)**(1/$N)) - $pi/2) * $degs;
                                    $longi = (($theta/$N) * $degs) + $lambda0;
    
                                    $resultant = sqrt($uwind[$i]**2 + $vwind[$i]**2);
                                    $angle = (atan2($vwind[$i],$uwind[$i])) * $degs;
                                            if ($angle<0) {
                                            $angle=$angle + 360;
                                           }
    
                                    $variance_speed = 0;
                                    $variance_angle = 0;
    
                                    $sth->execute ($lat, $longi, $x, $y, $resultant, $variance_speed, $angle, $variance_angle) || die "Couldn't insert record : $DBI::errstr";
    
    
     
    Last edited by a moderator: Aug 4, 2009
  2. nimesh

    nimesh New Member

    Joined:
    Apr 13, 2009
    Messages:
    770
    Likes Received:
    20
    Trophy Points:
    0
    Occupation:
    Oracle Apps Admin
    Location:
    Mumbai
    Home Page:
    I just attended this seminar yesterday and today I'm recommending it. COOL :)

    Try this one, Oracle Data Integrator.
    This just suites your requirement.

    1] Load data from Text files to MySQL
    2] Load tons of data in min amount of time

    The presenter had loaded data from a 30GB file in around 30 minutes.

    http://www.oracle.com/technology/software/products/odi/index.html
     

Share This Page