1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Help with httpwebrequest POST data to https

Discussion in 'C#' started by justJ, Feb 6, 2010.

  1. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    My client (real estate company owner) needs a program that he can install on his secretarys computer that will log into a web site to check times. If a certain time (4 days) has elapsed then I have to make a full screen WebBrowser which goes to that website and it will not go away until his secretary does the posts she has to do. That page is craigslist, now I can auto login with the WebBrowser control but I was looking for a more stealthy way about it and because I didn't want the WebBrowser to keep refreshing and eat up memory so I was looking into sending POST data through httpwebrequest / httpwebresponse. Slap me for making it 100 times harder on myself but I'm already in to deep and didn't think it would be this much of a pain to get working and I don't like to get beat.

    This is what I have:

    well it wouldn't let me paste the code because I got...

    The following errors occurred with your submission:
    Too many live links/images found in your post content. Please edit your post or contact the administrator.

    but if you download the .txt attachment then you can see it.

    View attachment hmm.txt


    All I can get in return is 2 ansi characters. What am I doing wrong?
     
  2. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    I knew i should have waited to ask for help, I needed to decompress the gzip stream. Sooooo happpy now!!
     
  3. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    couldn't find the Edit button...

    "I knew I shouldn't* have asked for help"...

    Figured it out this morning
     
  4. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    just out of curiosity, I tried to take it further. (My clients app is done btw and he will get it monday) I actually tried to post to craigslist with httpwebreqeust/response. I tackled the daunting task of finding all of the hidden field names/values and the random names given to other non hidden fields and my post string looks like..


    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="postingKey"

    ecTZogUM4km0OZ9l
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="test"

    k4EeW
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMCWF8JdisqDKvbaCEArAjjg9bBOlGgWCzWqdrC5s46N1T3vQSEiAdfg"

    U2FsdGVkX182MjMwNjIzMBRUyB2ookEk94PKQo1Xm5XmSiZVEjuAqhYgf6GfjFkT
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwN:jIz.M.C0lAgzH0cu_ViykKEboT.u4Y_sHlSRDS9_xKVoLqFYcieFptb7S_ejw"

    This is my title
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U:2FsdGVkX182MjMwNjIzMI2b9JMWF:w:MIOvBrhDf841qq0wPVt0SG.jw"

    100.00
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMNyAWgJxg2F6U561bgcedPnkV49.l2ymU1dzmUo6e5ZVdg3tD5tngSOY"

    westside
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="FromEMail"

    myEmail@gmail.com
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMHK928d5JVewkOa:zPnjduF:gsVY8jl6:DcEuxTJzcQAlrT"

    A
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMPAtotst_q9PxC3RKRnOno4fqbXZiK0rG-rXzS_HWzqF_efy2zyQIhc"

    U2FsdGVkX182MjMwNjIzMIDcFNP1axrTB6TDnoN2A1oradO0XTZVOgivLRn7M3IS4kqU7zDR90U
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMKGsCrxyEiINL90gcQUpoo.0NR61mHoLJ-Yg-G_:3Vu2QN"

    this is my big description!!!
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="image1"

    add
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="imgfile1"; filename=""
    Content-Type: application/octet-stream


    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="image2"

    add
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="imgfile2"; filename=""
    Content-Type: application/octet-stream


    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="image3"

    add
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="imgfile3"; filename=""
    Content-Type: application/octet-stream


    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="image4"

    add
    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="imgfile4"; filename=""
    Content-Type: application/octet-stream


    -----------------------------8cc755c93940630
    Content-Disposition: form-data; name="U2FsdGVkX182MjMwNjIzMB:RUyB2ookEk9k21dz4PVo_7ZmpNhqCsHLvxlI8jdEA6"

    Continue
    -----------------------------8cc755c93940630--


    Now, compared to what Fiddler show when I do an actual post, this is exactly the same. Here is my code to submit the information..

    Code:
    
                string myData = buildPostString(content);
                string url = getURL(content);
                string referer = urlReferer(url);
                ASCIIEncoding encoding = new ASCIIEncoding();
                //HttpWebRequest webReq = new HttpWebRequest(WebRequest.Create(getURL()), HttpWebRequest);
                byte[] buffer = encoding.GetBytes(myData);
                //tried the below buffer too..
                //byte[] buffer = System.Text.Encoding.GetEncoding(1251).GetBytes(myData);
                HttpWebRequest myRequest = WebRequest.Create(url) as HttpWebRequest;
                myRequest.Method = "POST";
                myRequest.KeepAlive = true;
                myRequest.AllowAutoRedirect = true;
                myRequest.CookieContainer = cookie;
                myRequest.ContentType = "multipart/form-data; boundary=" + boundary;
                myRequest.ContentLength = buffer.Length;
                myRequest.UserAgent = "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 6.1; WOW64; Trident/4.0; SLCC2; .NET CLR 2.0.50727; .NET CLR 3.5.30729; .NET CLR 3.0.30729; Media Center PC 6.0)";
                myRequest.Headers.Set("Accept-Language", "en-us");
                myRequest.Headers.Set("Accept-Encoding", "gzip, deflate");
                myRequest.ProtocolVersion = HttpVersion.Version10;
                myRequest.Accept = "application/x-ms-application, image/jpeg, application/xaml+xml, image/gif, image/pjpeg, application/x-ms-xbap, application/x-shockwave-flash, application/msword, */*";
                myRequest.Referer = url;
                myRequest.KeepAlive = true;
                myRequest.Headers.Add(HttpRequestHeader.CacheControl, "no-cache");
    
                myRequest.AllowWriteStreamBuffering = true;
    
                Stream newStream = myRequest.GetRequestStream();
    
                newStream.Write(buffer, 0, buffer.Length);
    
                newStream.Close();
                HttpWebResponse webResp = (HttpWebResponse)myRequest.GetResponse();
                myRequest.CookieContainer.Add(webResp.Cookies);
                Stream responseStream = responseStream = webResp.GetResponseStream();
                if (webResp.ContentEncoding.ToLower().Contains("gzip"))
                    responseStream = new System.IO.Compression.GZipStream(responseStream, System.IO.Compression.CompressionMode.Decompress);
                else if (webResp.ContentEncoding.ToLower().Contains("deflate"))
                    responseStream = new System.IO.Compression.DeflateStream(responseStream, System.IO.Compression.CompressionMode.Decompress);
    
                StreamReader responseReader = new StreamReader(responseStream, Encoding.Default);
    
                string result = responseReader.ReadToEnd();
                
                webResp.Close();
    
                return result;
    
    
    if I capture what my app is doing in fiddler and compare it to an actual manual post, they both have everything similar in the header and POST string. The page i get back in the response (in my app) is the same form that I want to fill out but it is empty. Instead I should get the next page after submitting the form which is "this is how your post will look" with a continue button. What am I doing wrong? I really hope I figure this out before someone else helps me... but don't let that stall you from helping me ;)
     
  5. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
  6. shabbir

    shabbir Administrator Staff Member

    Joined:
    Jul 12, 2004
    Messages:
    15,285
    Likes Received:
    364
    Trophy Points:
    83
    Made the link clickable.
     
  7. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    how you find the random name/value of non hidden fields.
    actualy i m posting(httpWebReques) data to webpage where the textboxes name are changing whenever i refresh the page.
    how i can find the name of those textboxes before posting the data.
     
  8. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    You have to scrape the HTML code from the web page, iterate through each <input> tag and apply the proper value to the random name. Probably the hardest part to make because the hidden fields with random names and values have to be dealt with too.

    Good luck.
     
  9. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    i have used to two httpWebRequest
    in the first request i scrap the page and take the name/value of hidden and no hidden fields.
    and in the second request i post those field names with proper values, but it not work, am i missing something here.....

    Thanks for any suggestions.
     
  10. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    probably need to fix your WebRequest headers, cookies ect... I little error and it will not work.
     
  11. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    Can you at least log into the webpage you are looking to automate? That is usually the easiest part and once you do that then the rest should follow. Don't forget to save cookies of each response and reuse them for the next request.
     
  12. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    this is how im doing it....
    plz let me know what i doing wrong here.

    Thanx in advace.
    ////////////////////////////////////////////////////////////////
     

    Attached Files:

  13. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    You thing you are doing wrong is not checking what page you got in return. You shouldn't make your login and your post form in the same method. Make a controller that keeps the string that the page returned, checks it against pages that could be from an error (you typed password wrong, you need to enter captcha, 4o4 file not found...) but mainly to check if it is on the page that you want. Example, have it read the page returned string for keep words. You don't need to add cookies like that (add.headers("cookie...)), just use webRequest.CookieContainer = CookieContainer and in your response you need to add the cookies you get back to your cookieContainer and pass that cookie container to be reused with all of your future request, until you change logins. Set your protocolVersion, cachecontrol, accepted language, encoding ect... Basically, everything that you see Fiddler do, you need to do the same thing in your post. If you don't know what Fiddler is then google it.

    Just focus on logging into the page first and sending the webPage string to a webBrowser control like "webBrowser1.DocumentText = webPageString;" Then just keep trying to login until webBrowser1 shows the web page that you are looking to land on.

    I'm not going to give out to much because that will bring me faster future competition to my auto poster that I'm about to make a paid members only poster. It's been free for 2 long and will only be free for the next couple of weeks.
     
  14. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    Thax for your precious advice.
    Now it z working....
    plz tell me one thing who to handle the captcha......
     
  15. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    I can't do that. I don't want to compromise the way that I do it because reCap can easily fix it.
     
  16. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    i used OCR but it is not working perfectly, coz it can't read some complex images. it work with other captcha's but not with recaptcha.com.
    if you have any solution plz guide me.
    You hav helped me alot and i m very thankful to you. If you provide me little assistane on captcha that vl be great.
    Thanx and looking for your positive reply.
     
  17. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    use a captcha solving company. All you have to do is upload the picture to them and then they send back the answer.
     
  18. tahir_123

    tahir_123 New Member

    Joined:
    Mar 31, 2010
    Messages:
    6
    Likes Received:
    0
    Trophy Points:
    0
    i m realy struggling to solve the captcha problem. i hv tried differen options , search alot but all in vain.
    plz help me to solve this problem.

    looking for your positive responce.
    thnx.
     
  19. justJ

    justJ New Member

    Joined:
    Feb 6, 2010
    Messages:
    12
    Likes Received:
    0
    Trophy Points:
    0
    You know how to use Request/Response and Decaptcher.com has way for you to submit the image via web form. If you go to their web site they show you exactly how to create the form and all you have to do is mimic that for your request and then the response will be the answer. You will have to do some string.Split('|'); array to divide the response up because they send back more info than just the answer.
     

Share This Page