1. This site uses cookies. By continuing to use this site, you are agreeing to our use of cookies. Learn More.

Java HTTP Wrapper

Discussion in 'Code Snippets and Tutorials' started by Zer0, Jul 28, 2008.

  1. Zer0

    Zer0 Level IV

    Joined:
    Mar 2, 2008
    Messages:
    3,037
    Likes Received:
    180
    Location:
    Home sweet home
    Here's a nice HTTP wrapper that I wrote and use in my programs. Its simple and very effective.

    Connnection.java
    Code (Text):
    1.  
    2. import java.io.*;
    3. import java.net.*;
    4. import java.util.*;
    5. import java.util.zip.*;
    6. import java.awt.image.BufferedImage;
    7. import javax.imageio.ImageIO;
    8.  
    9. public class Connection implements Serializable {
    10.     static final long serialVersionUID = 1L;
    11.    
    12.     String domain, referer;
    13.     Map<String,String> cookies;
    14.    
    15.     static String rpUseragent = "Mozilla/5.0 (X11; U; Linux x86_64; en-US; rv:1.8.1.14) Gecko/20080509 Firefox/2.0.0.14";
    16.     static String rpAcceptText = "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8";
    17.     static String rpAcceptPng = "image/png,image/*;q=0.8,*/*;q=0.5";
    18.     static String rpAcceptLanguage = "en-us,en;q=0.5";
    19.     static String rpAcceptEncoding = "gzip, deflate";
    20.     static String rpAcceptCharset = "ISO-8859-1,utf-8;q=0.7,*;q=0.7";
    21.     static String rpKeepAlive = "300";
    22.     static String rpConnection = "keep-alive";
    23.     static String rpContentType = "application/x-www-form-urlencoded";
    24.    
    25.     public Connection( String domain, Map<String,String> cookies, String referer ) {
    26.         this.domain = domain;
    27.         this.cookies = cookies;
    28.         this.referer = referer;
    29.     }
    30.    
    31.     public Connection( String domain, Map<String,String> cookies ) {
    32.         this( domain, cookies, null );
    33.     }
    34.    
    35.     public Connection( String domain, String referer ) {
    36.         this( domain, new HashMap<String,String>(), referer );
    37.     }
    38.    
    39.     public Connection( String domain ) {
    40.         this( domain, new HashMap<String,String>(), null );
    41.     }
    42.    
    43.     public String get( String url ) {
    44.         if( url.charAt( 0 ) == '/' )
    45.             url = domain + url;
    46.        
    47.         try {
    48.             HttpURLConnection conn = (HttpURLConnection)( new URL( url.replaceAll( " ", "%20" ) ).openConnection() );
    49.             setRequestProperties( conn );
    50.             conn.setRequestMethod( "GET" );
    51.             referer = url;
    52.             return read( conn );
    53.         } catch( IOException e1 ) {
    54.             e1.printStackTrace();
    55.             return null;
    56.         }
    57.     }
    58.    
    59.     public String post( String url, String[][] data ) {
    60.         if( url.charAt( 0 ) == '/' )
    61.             url = domain + url;
    62.        
    63.         try {
    64.             HttpURLConnection conn = (HttpURLConnection)( new URL( url.replaceAll( " ", "%20" ) ).openConnection() );
    65.             setRequestProperties( conn );
    66.             conn.setRequestMethod( "POST" );
    67.             conn.setDoOutput( true );
    68.            
    69.             StringBuilder sb = new StringBuilder();
    70.            
    71.             for( int i = 0; i < data[0].length; i++ )
    72.                 sb.append( URLEncoder.encode( data[0][i], "UTF-8" ) ).append( '=' ).append( URLEncoder.encode( data[1][i], "UTF-8" ) ).append( '&' );
    73.            
    74.             conn.setRequestProperty( "Content-Type", rpContentType );
    75.             conn.setRequestProperty( "Content-Length", Integer.toString( sb.length()-1 ) );
    76.            
    77.             PrintWriter out = new PrintWriter( new BufferedWriter( new OutputStreamWriter( conn.getOutputStream() ) ) );
    78.             out.write( sb.substring( 0, sb.length()-1 ) );
    79.             out.close();
    80.            
    81.             referer = url;
    82.             return read( conn );
    83.         } catch( IOException e1 ) {
    84.             e1.printStackTrace();
    85.             return null;
    86.         }
    87.     }
    88.    
    89.     public BufferedImage getImage( String url ) {
    90.         try {
    91.             HttpURLConnection conn = (HttpURLConnection)( new URL( ( url.charAt( 0 ) == '/' ? domain+url : url ).replaceAll( " ", "%20" ) ).openConnection() );
    92.             setRequestProperties( conn );
    93.             conn.setRequestMethod( "GET" );
    94.             conn.setRequestProperty( "Accept", rpAcceptPng );
    95.             return ImageIO.read( conn.getInputStream() );
    96.         } catch( IOException e1 ) {
    97.             e1.printStackTrace();
    98.             return null;
    99.         }
    100.     }
    101.    
    102.     public boolean hasCookie( String key ) {
    103.         return cookies.containsKey( key );
    104.     }
    105.    
    106.     public String getCookieString() {
    107.         StringBuilder sb = new StringBuilder();
    108.        
    109.         for( String s : cookies.keySet() )
    110.             sb.append( s ).append( '=' ).append( cookies.get( s ) ).append( ';' );
    111.        
    112.         return sb.toString();
    113.     }
    114.    
    115.     private void setRequestProperties( HttpURLConnection conn ) {
    116.         conn.setInstanceFollowRedirects( false );
    117.         conn.setRequestProperty( "User-Agent", rpUseragent );
    118.         conn.setRequestProperty( "Accept", rpAcceptText );
    119.         conn.setRequestProperty( "Accept-Language", rpAcceptLanguage );
    120.         conn.setRequestProperty( "Accept-Encoding", rpAcceptEncoding );
    121.         conn.setRequestProperty( "Accept-Charset", rpAcceptCharset );
    122.         conn.setRequestProperty( "Keep-Alive", rpKeepAlive );
    123.         conn.setRequestProperty( "Connection", rpConnection );
    124.        
    125.         if( referer != null && referer.length() != 0 )
    126.             conn.setRequestProperty( "Referer", referer );
    127.        
    128.         if( cookies != null && cookies.size() != 0 )
    129.             conn.setRequestProperty( "Cookie", getCookieString() );
    130.     }
    131.    
    132.     private String read( HttpURLConnection conn ) throws IOException {
    133.         BufferedReader in = null;
    134.        
    135.         if( conn.getContentEncoding() == null )
    136.             in = new BufferedReader( new InputStreamReader( conn.getInputStream() ) );
    137.         else
    138.             if( conn.getContentEncoding().equalsIgnoreCase( "gzip" ) )
    139.                 in = new BufferedReader( new InputStreamReader( new GZIPInputStream( conn.getInputStream() ) ) );
    140.             else if( conn.getContentEncoding().equalsIgnoreCase( "deflate" ) )
    141.                 in = new BufferedReader( new InputStreamReader( new InflaterInputStream( conn.getInputStream(), new Inflater( true ) ) ) );
    142.        
    143.         StringBuilder sb = new StringBuilder();
    144.         String s;
    145.        
    146.         while( ( s = in.readLine() ) != null )
    147.             sb.append( s ).append( '\n' );
    148.        
    149.         putCookies( conn.getHeaderFields().get( "Set-Cookie" ) );
    150.         return sb.toString();
    151.     }
    152.    
    153.     private void putCookies( List<String> cookieList ) {
    154.         if( cookieList == null )
    155.             return;
    156.        
    157.         int index;
    158.        
    159.         for( String cookie : cookieList )
    160.             cookies.put( cookie.substring( 0, index = cookie.indexOf( '=' ) ), cookie.substring( index+1, cookie.indexOf( ';', index ) ) );
    161.     }
    162. }
    163.  
    Here's how to use it:
    First you instantiate an instance of the Connection class like so
    Code (Text):
    1. Connection conn = new Connection( "http://www.neopets.com" );
    Now, you can use a number of methods in the Connection class.

    To make a GET request, use the get( url ) method:
    Code (Text):
    1. conn.get( "/bank.phtml" )
    Notice that you don't have to type the full URL, the domain name specified in the constructor is automatically filled in.
    If you do specify a domain however in the URL, it will go to that domain w/o the default domain filled in.

    POST requests work much the same way:
    Code (Text):
    1. conn.post( "/login.phtml", new String[][]{ { "username", "password", "destination" }, { user, password, "/" } } );
    This is how you log into Neopets using my wrapper. Just one simple line.
    The POST data is stored as a 2D array. The first array is the POST attribute, the second array holds the POST data corresponding to each attribute.
    You may also POST using just a String, but I've never used that myself explicitly, so I won't cover that here.

    getImage( url ) returns an image at the given URL. This is useful for making things like ONRs.
    Code (Text):
    1. conn.getImage( "/captcha_show.phtml?_x_pwned=" + hash )
    I find this wrapper particularly useful because you do not have to deal with referer URLs and cookies. It automatically handles both of these. It stores the previously accessed URL into a referer variable and uses that as the referer URL in the next request (see Connection.java for details). Cookies are also automatically updated, so there is no need for manually handling them.

    Enjoy and happy programming :)
     
    Kaden, junkant and chelsea1 like this.
  2. chelsea1

    chelsea1 Level IV

    Joined:
    Nov 26, 2006
    Messages:
    2,538
    Likes Received:
    31
    Location:
    London
    +rep top java example

    just one point, it has been advised that you don't use a one line login

    sorry if you have a multiple line one coded into the wrapper, i haven't used
     
  3. Zer0

    Zer0 Level IV

    Joined:
    Mar 2, 2008
    Messages:
    3,037
    Likes Received:
    180
    Location:
    Home sweet home
    Yes, I just used that one particular line in the example to show how a POST request could be done with this wrapper. In fact, it wouldn't work in one line with my wrapper because otherwise it wouldn't handle the referrer URL correctly. This wrapper forces the programmer to have the program go to the correct pages in order to work. I prefer to think of each instance of a Connection as a tab or a window on Neopets. Just make sure that you use the same CookieManager for each of the Connections.

    Here's a full log-in example:
    Code (Text):
    1. conn.post( "/hi.phtml", new String[][]{ { "destination", "username" }, { "/", user } } );
    2. String page = conn.post( "/login.phtml", new String[][]{ { "username", "password", "destination" }, { user, password, "/" } } );
    3. // code to determine if account is frozen, etc
     
  4. chelsea1

    chelsea1 Level IV

    Joined:
    Nov 26, 2006
    Messages:
    2,538
    Likes Received:
    31
    Location:
    London
    you couldn't post a code snippet with a complete class (ie one which works without any coding) where all it does is login could you (bogus usernames and passwords obviously)?

    thanks
     
  5. Zer0

    Zer0 Level IV

    Joined:
    Mar 2, 2008
    Messages:
    3,037
    Likes Received:
    180
    Location:
    Home sweet home
    Updated my wrapper to the latest version